-- Logs begin at Thu 2022-10-27 00:44:14 UTC, end at Thu 2022-10-27 01:54:14 UTC. -- Oct 27 01:10:20.815174 dv-cinder-815582-19-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Oct 27 01:10:20.824468 dv-cinder-815582-19-iscsi devstack@keystone.service[81575]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Oct 27 01:10:20.826466 dv-cinder-815582-19-iscsi devstack@keystone.service[81575]: *** Starting uWSGI 2.0.18-debian (64bit) on [Thu Oct 27 01:10:20 2022] *** Oct 27 01:10:20.826466 dv-cinder-815582-19-iscsi devstack@keystone.service[81575]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Oct 27 01:10:20.826466 dv-cinder-815582-19-iscsi devstack@keystone.service[81575]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Oct 27 01:10:20.826466 dv-cinder-815582-19-iscsi devstack@keystone.service[81575]: nodename: dv-cinder-815582-19-iscsi Oct 27 01:10:20.826466 dv-cinder-815582-19-iscsi devstack@keystone.service[81575]: machine: x86_64 Oct 27 01:10:20.826466 dv-cinder-815582-19-iscsi devstack@keystone.service[81575]: clock source: unix Oct 27 01:10:20.826466 dv-cinder-815582-19-iscsi devstack@keystone.service[81575]: pcre jit disabled Oct 27 01:10:20.826466 dv-cinder-815582-19-iscsi devstack@keystone.service[81575]: detected number of CPU cores: 4 Oct 27 01:10:20.826466 dv-cinder-815582-19-iscsi devstack@keystone.service[81575]: current working directory: / Oct 27 01:10:20.826466 dv-cinder-815582-19-iscsi devstack@keystone.service[81575]: detected binary path: /usr/bin/uwsgi-core Oct 27 01:10:20.826466 dv-cinder-815582-19-iscsi devstack@keystone.service[81575]: your processes number limit is 47680 Oct 27 01:10:20.826466 dv-cinder-815582-19-iscsi devstack@keystone.service[81575]: your memory page size is 4096 bytes Oct 27 01:10:20.827125 dv-cinder-815582-19-iscsi devstack@keystone.service[81575]: detected max file descriptor number: 2048 Oct 27 01:10:20.827125 dv-cinder-815582-19-iscsi devstack@keystone.service[81575]: lock engine: pthread robust mutexes Oct 27 01:10:20.827125 dv-cinder-815582-19-iscsi devstack@keystone.service[81575]: thunder lock: enabled Oct 27 01:10:20.827125 dv-cinder-815582-19-iscsi devstack@keystone.service[81575]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Oct 27 01:10:20.827125 dv-cinder-815582-19-iscsi devstack@keystone.service[81575]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Oct 27 01:10:20.866218 dv-cinder-815582-19-iscsi devstack@keystone.service[81575]: Python main interpreter initialized at 0x5595039169c0 Oct 27 01:10:20.866218 dv-cinder-815582-19-iscsi devstack@keystone.service[81575]: python threads support enabled Oct 27 01:10:20.866218 dv-cinder-815582-19-iscsi devstack@keystone.service[81575]: your server socket listen backlog is limited to 100 connections Oct 27 01:10:20.866218 dv-cinder-815582-19-iscsi devstack@keystone.service[81575]: your mercy for graceful operations on workers is 90 seconds Oct 27 01:10:20.866586 dv-cinder-815582-19-iscsi devstack@keystone.service[81575]: mapped 403077 bytes (393 KB) for 2 cores Oct 27 01:10:20.866667 dv-cinder-815582-19-iscsi devstack@keystone.service[81575]: *** Operational MODE: preforking *** Oct 27 01:10:20.866889 dv-cinder-815582-19-iscsi devstack@keystone.service[81575]: *** uWSGI is running in multiple interpreter mode *** Oct 27 01:10:20.866889 dv-cinder-815582-19-iscsi devstack@keystone.service[81575]: spawned uWSGI master process (pid: 81575) Oct 27 01:10:20.867087 dv-cinder-815582-19-iscsi systemd[1]: Started Devstack devstack@keystone.service. Oct 27 01:10:20.868161 dv-cinder-815582-19-iscsi devstack@keystone.service[81575]: spawned uWSGI worker 1 (pid: 81587, cores: 1) Oct 27 01:10:20.868161 dv-cinder-815582-19-iscsi devstack@keystone.service[81575]: spawned uWSGI worker 2 (pid: 81588, cores: 1) Oct 27 01:10:20.868161 dv-cinder-815582-19-iscsi devstack@keystone.service[81575]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Oct 27 01:10:25.183356 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 27 01:10:25.183356 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: from cryptography.utils import int_from_bytes Oct 27 01:10:25.184117 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 27 01:10:25.184117 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: from cryptography.utils import int_from_bytes Oct 27 01:10:25.184502 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 27 01:10:25.184502 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: from cryptography.utils import int_from_bytes Oct 27 01:10:25.185024 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 27 01:10:25.185024 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: from cryptography.utils import int_from_bytes Oct 27 01:10:25.235197 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 27 01:10:25.235387 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 27 01:10:25.235494 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 27 01:10:25.235590 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] command line args: [] {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 27 01:10:25.235665 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 27 01:10:25.235838 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 27 01:10:25.235931 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] command line args: [] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 27 01:10:25.236005 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ================================================================================ {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 27 01:10:25.236108 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 27 01:10:25.236256 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] admin_token = **** {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.236338 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ================================================================================ {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 27 01:10:25.236503 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] config_dir = [] {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.236596 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] admin_token = **** {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.236676 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.236749 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] config_dir = [] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.236858 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] config_source = [] {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.236944 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.237038 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] debug = True {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.237170 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] config_source = [] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.237252 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.237505 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] debug = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.237505 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.237646 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.237707 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.237769 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.237837 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.238027 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.238112 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.238276 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.238363 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.238508 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] list_limit = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.238603 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] list_limit = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.238663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] log_config_append = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.238725 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] log_config_append = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.238874 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.238961 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.239022 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] log_dir = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.239085 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] log_dir = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.239202 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] log_file = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.239304 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] log_file = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.239455 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.239544 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.239670 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.239748 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.240368 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.240453 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.240686 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.240865 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.241042 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.241143 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.241274 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.241363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.241445 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.241548 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.241667 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.241786 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.241916 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.242006 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.242151 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.242236 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.242384 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.242458 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.242617 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.242713 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.242849 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.242933 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.243096 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.243200 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.243338 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.243612 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.243735 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.42/identity {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.243855 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] publish_errors = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.244088 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.42/identity {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.244219 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.244297 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] publish_errors = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.244370 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.244481 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.244567 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.244705 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.244789 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.244882 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.244980 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.245147 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.245230 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] standard_threads = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.245382 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.245469 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.245613 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] standard_threads = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.245701 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.245805 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.245882 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.245981 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] use_journal = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.246059 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.246157 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] use_json = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.246343 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.246430 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] use_stderr = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.246504 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] use_journal = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.246581 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] use_syslog = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.246660 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] use_json = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.246733 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] use_stderr = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.246812 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.246885 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] use_syslog = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.246962 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.247074 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 27 01:10:25.247181 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.247327 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.247441 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.247563 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.247683 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.247821 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.247934 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.248063 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.248177 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.248309 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.248427 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.248546 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.248666 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.248790 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.248917 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.249022 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.249192 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.249280 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.249452 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.249577 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.249691 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.249773 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.249939 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.250025 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.250181 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.250277 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.250455 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.250584 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.250712 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.250846 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.250994 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.251124 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.251267 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.251396 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.251523 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.251650 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.251777 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.251978 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.252106 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.252240 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.252369 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.252506 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.252634 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.252773 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.252910 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.253039 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.253150 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.253226 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.253298 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.253380 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.253459 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.253533 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.253604 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.253675 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.253752 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.253833 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.253906 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.253977 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.254096 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.254176 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.254343 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.254430 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.254570 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.254648 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.254797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.254875 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.255041 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.255129 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.255285 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.255364 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.256143 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.256383 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.256511 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.256604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.256791 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.256871 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.257032 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.257137 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.257302 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.257393 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.257532 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.257609 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.257762 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.257846 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.257995 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.258145 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.258227 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.258301 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.258451 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.258530 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.258687 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.258778 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.258918 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.258995 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.259155 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.259233 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.259390 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.259476 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.259621 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.259719 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.259842 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.259947 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.260076 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.260190 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.260318 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.260422 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.260545 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.260649 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.260782 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.260862 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.261020 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.261161 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.261275 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.261357 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.261488 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.261572 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.261725 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.261805 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.261954 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.262043 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.262202 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.262287 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.262437 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.262528 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.262671 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.262751 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.262902 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.263001 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.263159 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.263241 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.263395 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.263478 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.263625 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.263716 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.263861 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.263947 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.264095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.264174 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.264359 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.264439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.264594 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.264675 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.264830 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.264908 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.265081 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.265167 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.265326 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.265410 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.265560 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.265641 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.265792 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.265872 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.266037 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.266117 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.266270 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.266349 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.266505 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.266587 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.266744 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.266824 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.266976 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.267055 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.267212 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.267291 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.267444 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.267524 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.267676 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.267756 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.267916 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.267997 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.268151 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.268232 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.268383 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.268469 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.268631 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.268717 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.268869 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.268949 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.269133 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.269214 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.269371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.269460 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.269611 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.269687 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.269836 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.269917 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.270068 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.270155 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.270298 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.270385 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.270528 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.270607 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.270756 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.270836 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.270983 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.271128 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.271266 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.271367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.271507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.271759 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.271994 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.272084 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.272236 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.272317 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.272484 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.272566 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.272723 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.272817 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.272954 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] auth.external = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.273039 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.273211 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.273294 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.273471 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.273560 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.273732 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.273820 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.273962 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] auth.password = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.274048 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.274190 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] auth.external = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.274270 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] auth.token = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.274419 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.274507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.274651 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.274736 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.274900 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.274981 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.275136 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.275224 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] auth.password = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.275363 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] auth.token = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.275443 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.275599 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.275679 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.275833 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.275926 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.276068 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] credential.caching = True {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.276147 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.276294 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.276373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.276539 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.276628 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.276784 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.276871 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.277012 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.277114 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.277269 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] credential.caching = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.277362 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.277514 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.277595 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.277758 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.277846 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.278011 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.278098 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.278251 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.278333 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.278492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.278573 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.278724 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.278813 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.278968 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.279049 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.279218 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.279300 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.279451 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.279540 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.279700 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.279788 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.279950 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] federation.caching = True {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.280029 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.280189 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.280278 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.280443 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.280523 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.280680 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.280759 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.280909 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.280989 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.281178 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.281262 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] federation.caching = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.281414 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.281493 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.281659 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.281741 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.281897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.281976 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.282122 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.282202 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.282367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.282449 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.282600 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.282681 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.282845 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] identity.caching = True {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.282924 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.283078 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.283159 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.283331 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.283411 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.283567 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.283646 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.283793 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.283872 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.284030 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] identity.caching = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.284109 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.284265 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.284344 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.284507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.284589 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.284737 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.284817 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.284969 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.285050 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.285242 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.285323 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.285471 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.285705 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.285791 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.285951 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.286041 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.286201 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.286281 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.286430 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.286512 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.286658 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.286761 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.286959 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.287280 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.287365 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.287571 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.287652 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.287813 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.287895 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.288055 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.288141 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.288289 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.288370 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.288528 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.288611 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.288761 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.288841 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.288998 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.289155 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.289265 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.289349 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.289492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.289572 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.289730 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.289819 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.289960 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.290039 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.290191 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.290276 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.290428 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.290517 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.290665 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.290746 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.290909 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.290991 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.291138 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.291226 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.291372 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.291451 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.291611 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.291691 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.291841 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.291928 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.292076 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.292155 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.292314 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.292394 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.292538 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.292665 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.292772 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.292912 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.292992 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.293176 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.293258 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.293434 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.293516 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.293686 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.293766 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.293919 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.293998 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.294150 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.294260 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.294401 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.294511 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.294653 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.294741 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.294893 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.294975 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.295133 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.295213 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.295372 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.295453 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.295607 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.user = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.295689 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.295850 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.295931 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.296098 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.296178 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.296332 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.296413 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.296558 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.user = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.296638 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.296796 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.296878 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.297051 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.297163 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.297301 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.297382 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.297532 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.297610 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.297770 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.297852 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.298006 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.298087 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.298237 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.298316 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.298465 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.298547 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.298698 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.298779 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.298929 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.299010 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.299163 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.299244 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.299389 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.299469 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.299631 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.299711 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.299870 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.299951 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.300109 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.300190 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.300351 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.300430 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.300590 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.300672 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.300813 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.300893 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.301051 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.301147 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.301308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.301389 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.301542 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.301623 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.301774 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.301855 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.302006 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.302088 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.302247 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.302836 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.303035 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.303311 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.303421 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.303500 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.303846 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.303928 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.304108 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.304189 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.304346 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.304429 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.304586 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.304667 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.304822 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.304902 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.305111 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.305198 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] resource.caching = True {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.305317 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.305399 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.305564 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.305905 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.306066 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.306147 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.306293 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.306374 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.306528 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.306616 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.306768 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] resource.caching = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.306849 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.307013 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.307098 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.307253 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.307341 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.307484 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.307564 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.307700 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] role.caching = True {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.307781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.307953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] role.driver = sql {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.308108 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.308186 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.308259 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.308413 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.308493 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.308652 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.308740 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.308887 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.308968 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.309140 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] role.caching = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.309225 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.309387 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] role.driver = sql {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.309480 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.309616 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.309697 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.309861 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.309988 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.310106 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.310195 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.310340 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.310421 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.310581 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.310662 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.310824 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.310913 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.311067 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.311148 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.311308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.311389 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.311549 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.311698 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.311781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.311937 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.312016 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.312188 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.312268 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.312438 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.312519 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.312689 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.312770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.312939 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.313112 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.313197 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.313333 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.313442 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.313568 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.313697 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.313813 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.313937 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.314058 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.314172 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.314302 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.314404 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.314533 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.314638 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.314784 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.314891 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.315033 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.315147 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.315268 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.315396 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.315497 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.315634 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.315714 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.315871 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.315963 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.316103 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.316185 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.316339 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] token.caching = True {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.316419 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.316578 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.316670 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.316827 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.316909 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.317081 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.317166 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.317315 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.317407 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.317546 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] token.caching = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.317627 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.317778 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.317859 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.318094 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.318207 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.318512 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.318762 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.318851 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.319002 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.319084 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.319242 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.319325 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.319477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.319558 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.319724 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.319805 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.320001 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.320083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.320234 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.320327 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.320473 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.320553 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.320711 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.320800 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.320942 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.321023 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] database.connection = **** {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.321208 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.321291 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.321480 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.321573 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.321711 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.321801 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.321948 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.322029 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.322182 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.322263 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.322420 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.322509 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.connection = **** {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.322657 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.322737 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.322893 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.322974 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.323129 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.323218 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.323364 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.323444 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.323593 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.323680 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.323827 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.323916 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.324061 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.324142 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.324293 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.324375 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.324534 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.324623 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.324763 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.324843 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.325000 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.325107 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.325258 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.325352 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.325430 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81588) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 27 01:10:25.325505 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.325722 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.325964 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.326191 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.326427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.326656 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 27 01:10:25.326823 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 27 01:10:25.355872 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81588) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.356099 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81588) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.356307 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.356395 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81588) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.356536 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.356760 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.364722 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81588) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.365504 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.394415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81588) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.394644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81588) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.394730 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.394870 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.448474 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81588) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.449354 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.497861 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81588) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.498093 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81588) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.498692 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.498923 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.654887 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81588) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.655113 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81588) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.655337 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81588) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.655441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.655559 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81588) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.655657 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.655778 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81588) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.655863 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.655996 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81588) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.656080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.656236 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.656450 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.679671 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81588) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.679902 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81588) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.680100 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.680210 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81588) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.680331 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.680418 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81588) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.680555 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.680886 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81588) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.681027 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.681210 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81588) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.681294 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.681369 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81588) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.681473 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.681685 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.692409 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.692714 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81588) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.706371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.706553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.706651 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81588) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.706776 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.706862 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81588) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.706994 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.707079 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81588) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.707208 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.707291 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81588) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.707419 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.707504 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81588) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.707629 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.707713 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81588) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.707881 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81588) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 27 01:10:25.725650 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.725858 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 27 01:10:25.726201 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.726285 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.726404 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 27 01:10:25.726504 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 27 01:10:25.726681 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.726765 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.726929 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 27 01:10:25.727013 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.727203 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.727449 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.727534 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.727715 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.727955 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.728217 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.737227 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.737510 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.737595 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.737786 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.737870 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.738072 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.738165 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.738368 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.745262 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81588) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 27 01:10:25.745973 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81587) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 27 01:10:25.748167 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81588) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 27 01:10:25.748456 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.748745 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.748853 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81587) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 27 01:10:25.749034 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.749163 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.749345 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.749429 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.749611 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.749714 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.749882 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.749996 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.750158 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.750241 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.750513 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.750594 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.750787 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.750878 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.751083 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.751166 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.751428 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.751697 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.774363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.775749 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.775834 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81588) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 27 01:10:25.776106 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.777222 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81587) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 27 01:10:25.777508 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.782526 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81588) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 27 01:10:25.782819 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.783104 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.783821 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81587) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 27 01:10:25.784107 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.784388 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.790116 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81588) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 27 01:10:25.790682 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.791877 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81587) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 27 01:10:25.792165 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.794891 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81588) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 27 01:10:25.795200 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.795506 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.795794 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.796081 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.796326 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81587) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 27 01:10:25.796419 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.796618 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.796702 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.796895 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.796976 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.797195 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.797542 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.797751 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.798030 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.798305 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.813139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.813452 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.814819 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.815101 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.817341 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81588) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 27 01:10:25.819925 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81587) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 27 01:10:25.821307 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.821611 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.823148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.823444 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.825888 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81588) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 27 01:10:25.827666 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81587) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 27 01:10:25.829050 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81588) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 27 01:10:25.830814 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81587) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 27 01:10:25.832322 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.832626 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.832918 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.833241 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.833528 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.834485 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.835048 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.835348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.835633 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.835934 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.836227 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.836509 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.850910 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81588) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 27 01:10:25.851192 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.851461 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.851728 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.852815 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81587) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 27 01:10:25.853125 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.853393 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.853654 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.858704 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.860249 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.860486 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.861846 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.862044 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.862221 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.863557 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.863847 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.865501 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81588) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 27 01:10:25.865819 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.866210 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.866503 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.866798 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.867550 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81587) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 27 01:10:25.867834 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.868219 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.868502 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.868782 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.878053 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81588) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 27 01:10:25.878342 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.878724 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.879013 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.879315 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.879610 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.879888 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.879971 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81587) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 27 01:10:25.880179 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.881635 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.881935 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.882230 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.882507 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.882795 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.896446 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81588) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 27 01:10:25.898803 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81587) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 27 01:10:25.899397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81588) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 27 01:10:25.901733 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81587) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 27 01:10:25.902368 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.903850 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.904657 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.905495 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81588) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 27 01:10:25.905798 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.906086 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.906169 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.907683 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81587) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 27 01:10:25.907968 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.908247 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.912589 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.914099 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81588) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 27 01:10:25.915133 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.916583 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81587) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 27 01:10:25.917157 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.917454 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.917745 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.918024 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.919577 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.919863 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.920142 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.920412 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.925923 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81588) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 27 01:10:25.926219 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.926513 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.929210 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81587) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 27 01:10:25.929501 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.929782 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.935414 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81588) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 27 01:10:25.935705 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.935990 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.936272 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.936550 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.936826 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.937133 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.937425 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.937710 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.937821 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81587) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 27 01:10:25.937982 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.938107 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.938256 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.938388 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.938538 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.938662 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.938816 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.938933 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.939080 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81588) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.939201 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.939472 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.939758 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.940026 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.940307 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.940620 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.940895 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.941180 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.941441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 27 01:10:25.972954 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x5595039169c0 pid: 81588 (default app) Oct 27 01:10:25.976296 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x5595039169c0 pid: 81587 (default app) Oct 27 01:10:25.976389 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735ca4de-7560-473f-8852-81f769712b3e None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:25.976664 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735ca4de-7560-473f-8852-81f769712b3e None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:25.976941 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735ca4de-7560-473f-8852-81f769712b3e None None] PATH_INFO: `/v3` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:25.981152 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1/1] 10.222.0.42 () {50 vars in 791 bytes} [Thu Oct 27 01:10:25 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:10:32.855306 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebb5216-04ec-4ffd-b4a4-a4b3c3627b5a None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:32.855773 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebb5216-04ec-4ffd-b4a4-a4b3c3627b5a None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:32.855861 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebb5216-04ec-4ffd-b4a4-a4b3c3627b5a None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:32.860154 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1/2] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:10:32 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 27 01:10:32.866153 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4db71cd-066b-40de-8583-82e3b64efa77 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:32.866674 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4db71cd-066b-40de-8583-82e3b64efa77 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:32.866881 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4db71cd-066b-40de-8583-82e3b64efa77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:32.938375 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: INFO dbcounter [None req-f4db71cd-066b-40de-8583-82e3b64efa77 None None] Registered counter for database keystone Oct 27 01:10:32.957724 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG oslo_db.sqlalchemy.engines [None req-f4db71cd-066b-40de-8583-82e3b64efa77 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81588) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:326}} Oct 27 01:10:32.963785 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writer thread running {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 27 01:10:33.139634 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG passlib.handlers.bcrypt [None req-f4db71cd-066b-40de-8583-82e3b64efa77 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81588) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 27 01:10:33.139935 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG passlib.handlers.bcrypt [None req-f4db71cd-066b-40de-8583-82e3b64efa77 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81588) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 27 01:10:33.198025 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-f4db71cd-066b-40de-8583-82e3b64efa77 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:10:33.199615 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-f4db71cd-066b-40de-8583-82e3b64efa77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:10:33.325533 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 2/3] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:10:32 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 461 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 27 01:10:33.779076 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ebb5216-04ec-4ffd-b4a4-a4b3c3627b5a None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:33.848813 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: INFO dbcounter [None req-5ebb5216-04ec-4ffd-b4a4-a4b3c3627b5a None None] Registered counter for database keystone Oct 27 01:10:33.870396 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG oslo_db.sqlalchemy.engines [None req-5ebb5216-04ec-4ffd-b4a4-a4b3c3627b5a None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81587) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:326}} Oct 27 01:10:33.876552 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writer thread running {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 27 01:10:33.995739 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ebb5216-04ec-4ffd-b4a4-a4b3c3627b5a None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:33.999995 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b7d3af-8fbb-4007-b062-7375028a50aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:34.000610 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b7d3af-8fbb-4007-b062-7375028a50aa admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:34.000888 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b7d3af-8fbb-4007-b062-7375028a50aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:34.001189 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b7d3af-8fbb-4007-b062-7375028a50aa admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:34.009920 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b7d3af-8fbb-4007-b062-7375028a50aa admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:34.061987 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: WARNING keystone.common.rbac_enforcer.enforcer [None req-89b7d3af-8fbb-4007-b062-7375028a50aa admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 27 01:10:34.062348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b7d3af-8fbb-4007-b062-7375028a50aa admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:34.072080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 2/4] 10.222.0.42 () {58 vars in 1324 bytes} [Thu Oct 27 01:10:33 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 27 01:10:34.078012 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4db71cd-066b-40de-8583-82e3b64efa77 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:34.094961 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4db71cd-066b-40de-8583-82e3b64efa77 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:34.099409 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcd9af8-99e8-456b-8dc1-dd9408ce43ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:34.100507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcd9af8-99e8-456b-8dc1-dd9408ce43ea admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:34.100789 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcd9af8-99e8-456b-8dc1-dd9408ce43ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:34.101078 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcd9af8-99e8-456b-8dc1-dd9408ce43ea admin admin] PATH_INFO: `/v3/projects/062e383759c34ca3b52c150dd337a6ba` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:34.104840 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdcd9af8-99e8-456b-8dc1-dd9408ce43ea admin admin] RBAC: Authorizing `identity:get_project(project_id=062e383759c34ca3b52c150dd337a6ba)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:34.155015 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: WARNING keystone.common.rbac_enforcer.enforcer [None req-fdcd9af8-99e8-456b-8dc1-dd9408ce43ea admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 27 01:10:34.155360 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdcd9af8-99e8-456b-8dc1-dd9408ce43ea admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:34.159826 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 3/5] 10.222.0.42 () {56 vars in 1207 bytes} [Thu Oct 27 01:10:34 2022] GET /identity/v3/projects/062e383759c34ca3b52c150dd337a6ba => generated 344 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:10:36.273391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29b7be7-d963-4ccf-bf0f-a1cef4d1bfe5 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:36.273910 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29b7be7-d963-4ccf-bf0f-a1cef4d1bfe5 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:36.273910 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29b7be7-d963-4ccf-bf0f-a1cef4d1bfe5 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:36.277988 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 3/6] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:10:36 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 27 01:10:36.283426 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb39cb2-347c-4a27-9bde-ffb6c18d10ca None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:36.283788 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb39cb2-347c-4a27-9bde-ffb6c18d10ca None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:36.283968 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb39cb2-347c-4a27-9bde-ffb6c18d10ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:36.304278 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-5cb39cb2-347c-4a27-9bde-ffb6c18d10ca None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:10:36.305816 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-5cb39cb2-347c-4a27-9bde-ffb6c18d10ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:10:36.313722 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 4/7] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:10:36 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 27 01:10:36.587510 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29b7be7-d963-4ccf-bf0f-a1cef4d1bfe5 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:36.600174 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29b7be7-d963-4ccf-bf0f-a1cef4d1bfe5 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:36.604931 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de02a225-37bb-43e4-b72e-2b73feaaea8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:36.605559 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de02a225-37bb-43e4-b72e-2b73feaaea8d admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:36.605837 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de02a225-37bb-43e4-b72e-2b73feaaea8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:36.606106 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de02a225-37bb-43e4-b72e-2b73feaaea8d admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:36.613768 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de02a225-37bb-43e4-b72e-2b73feaaea8d admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:36.616092 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de02a225-37bb-43e4-b72e-2b73feaaea8d admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:36.624658 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 4/8] 10.222.0.42 () {58 vars in 1324 bytes} [Thu Oct 27 01:10:36 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 27 01:10:36.630067 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb39cb2-347c-4a27-9bde-ffb6c18d10ca None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:36.639492 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb39cb2-347c-4a27-9bde-ffb6c18d10ca None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:36.643708 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0213b9f4-ae38-4fe4-a7ab-523ea9514fc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:36.644317 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0213b9f4-ae38-4fe4-a7ab-523ea9514fc1 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:36.644591 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0213b9f4-ae38-4fe4-a7ab-523ea9514fc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:36.644860 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0213b9f4-ae38-4fe4-a7ab-523ea9514fc1 admin admin] PATH_INFO: `/v3/users/fe01809d18f644f9bc72669fc872b53d` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:36.647196 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0213b9f4-ae38-4fe4-a7ab-523ea9514fc1 admin admin] RBAC: Authorizing `identity:get_user(user_id=fe01809d18f644f9bc72669fc872b53d)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:36.649005 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0213b9f4-ae38-4fe4-a7ab-523ea9514fc1 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:36.650532 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 5/9] 10.222.0.42 () {56 vars in 1198 bytes} [Thu Oct 27 01:10:36 2022] GET /identity/v3/users/fe01809d18f644f9bc72669fc872b53d => generated 247 bytes in 21 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:10:38.851890 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b4e6d6-56ea-4236-8b47-ee8917175b09 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:38.852475 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b4e6d6-56ea-4236-8b47-ee8917175b09 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:38.852475 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b4e6d6-56ea-4236-8b47-ee8917175b09 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:38.853768 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 5/10] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:10:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:10:38.858866 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12772a30-07f5-4e28-a826-6fbe210e26df None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:38.859296 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12772a30-07f5-4e28-a826-6fbe210e26df None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:38.859480 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12772a30-07f5-4e28-a826-6fbe210e26df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:38.878292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-12772a30-07f5-4e28-a826-6fbe210e26df None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:10:38.879440 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-12772a30-07f5-4e28-a826-6fbe210e26df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:10:38.923102 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4a3cc8-c357-4bbc-ade4-31079d63348a None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:38.923381 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4a3cc8-c357-4bbc-ade4-31079d63348a None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:38.923654 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4a3cc8-c357-4bbc-ade4-31079d63348a None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:38.925006 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 6/11] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:10:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 27 01:10:38.930783 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8ff4e9-2d16-4077-93c6-2a013c6c0787 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:38.931385 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8ff4e9-2d16-4077-93c6-2a013c6c0787 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:38.931721 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8ff4e9-2d16-4077-93c6-2a013c6c0787 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:38.963050 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 6/12] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:10:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 27 01:10:38.973314 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG passlib.handlers.bcrypt [None req-db8ff4e9-2d16-4077-93c6-2a013c6c0787 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81587) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 27 01:10:38.973706 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG passlib.handlers.bcrypt [None req-db8ff4e9-2d16-4077-93c6-2a013c6c0787 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81587) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 27 01:10:39.023474 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-db8ff4e9-2d16-4077-93c6-2a013c6c0787 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:10:39.025017 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-db8ff4e9-2d16-4077-93c6-2a013c6c0787 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:10:39.122593 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 7/13] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:10:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 27 01:10:39.230380 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12772a30-07f5-4e28-a826-6fbe210e26df None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:39.254340 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:10:39.332585 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12772a30-07f5-4e28-a826-6fbe210e26df None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:39.366050 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c3ab0c-3d4c-44bb-93fb-2ac92a232f9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:39.366682 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c3ab0c-3d4c-44bb-93fb-2ac92a232f9c None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:39.366960 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c3ab0c-3d4c-44bb-93fb-2ac92a232f9c None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:39.367226 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c3ab0c-3d4c-44bb-93fb-2ac92a232f9c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:39.377958 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4c3ab0c-3d4c-44bb-93fb-2ac92a232f9c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:10:39.387361 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-a4c3ab0c-3d4c-44bb-93fb-2ac92a232f9c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:10:39.387361 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:10:39.387361 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:10:39.387361 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:10:39.387361 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:10:39.387361 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:10:39.387361 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:10:39.387361 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:10:39.387361 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:10:39.387361 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:10:39.387361 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:10:39.387361 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:10:39.387361 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:10:39.387361 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:10:39.387361 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:10:39.387361 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:10:39.388488 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:10:39.388488 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:10:39.388488 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:10:39.388488 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:10:39.388488 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:10:39.388488 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:10:39.388488 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:10:39.388488 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:10:39.388488 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:10:39.388488 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:10:39.388488 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:10:39.388488 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:10:39.388488 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:10:39.388488 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:10:39.388488 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:10:39.388488 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:10:39.389468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:10:39.389468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:10:39.389468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:10:39.389468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:10:39.389468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:10:39.389468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:10:39.389468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:10:39.389468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:10:39.389468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:10:39.389468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:10:39.389468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:10:39.389468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:10:39.389468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 7/14] 10.222.0.42 () {56 vars in 1096 bytes} [Thu Oct 27 01:10:39 2022] GET /identity/v3/roles/admin => generated 83 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:10:39.394238 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8ff4e9-2d16-4077-93c6-2a013c6c0787 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:39.395779 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c3ab0c-3d4c-44bb-93fb-2ac92a232f9c None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:39.476985 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:10:39.496516 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c3ab0c-3d4c-44bb-93fb-2ac92a232f9c None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:39.501112 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8ff4e9-2d16-4077-93c6-2a013c6c0787 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:39.503433 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:10:39.530794 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14ec615-3ac3-4bde-9397-66f8e123f5ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:39.531393 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14ec615-3ac3-4bde-9397-66f8e123f5ad None admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:39.531661 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14ec615-3ac3-4bde-9397-66f8e123f5ad None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:39.531930 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14ec615-3ac3-4bde-9397-66f8e123f5ad None admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:39.532591 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b14ec615-3ac3-4bde-9397-66f8e123f5ad None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:39.534065 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b14ec615-3ac3-4bde-9397-66f8e123f5ad None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:39.540348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e77f7241-81bd-4d47-ae60-4ca87b63b196 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:39.541254 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77f7241-81bd-4d47-ae60-4ca87b63b196 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:39.541541 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77f7241-81bd-4d47-ae60-4ca87b63b196 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:39.541859 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77f7241-81bd-4d47-ae60-4ca87b63b196 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:39.542901 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e77f7241-81bd-4d47-ae60-4ca87b63b196 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:10:39.543228 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e77f7241-81bd-4d47-ae60-4ca87b63b196 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:39.544265 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.sql.core [None req-b14ec615-3ac3-4bde-9397-66f8e123f5ad None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 27 01:10:39.544265 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 27 01:10:39.544265 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [parameters: {'id': '58d5548b0a61437eb9976a8eb7fc1eed', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 27 01:10:39.544265 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81588) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 27 01:10:39.544780 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e77f7241-81bd-4d47-ae60-4ca87b63b196 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:39.557503 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 8/15] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:10:39 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:10:39.562653 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-b14ec615-3ac3-4bde-9397-66f8e123f5ad None admin] Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 27 01:10:39.562653 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:10:39.562653 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 27 01:10:39.562653 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 27 01:10:39.562653 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 27 01:10:39.562653 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 27 01:10:39.562653 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 27 01:10:39.562653 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application result = self._query(query) Oct 27 01:10:39.562653 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 27 01:10:39.562653 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application conn.query(q) Oct 27 01:10:39.562653 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 27 01:10:39.562653 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 27 01:10:39.562653 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 27 01:10:39.562653 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application result.read() Oct 27 01:10:39.563599 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 27 01:10:39.563599 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 27 01:10:39.563599 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 27 01:10:39.563599 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 27 01:10:39.563599 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 27 01:10:39.563599 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 27 01:10:39.563599 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 27 01:10:39.563599 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 27 01:10:39.563599 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 27 01:10:39.563599 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:10:39.563599 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 27 01:10:39.563599 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:10:39.563599 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:10:39.563599 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 27 01:10:39.563599 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 27 01:10:39.563599 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 27 01:10:39.563599 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 27 01:10:39.563599 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 27 01:10:39.564516 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application next(self.gen) Oct 27 01:10:39.564516 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1068, in _transaction_scope Oct 27 01:10:39.564516 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application yield resource Oct 27 01:10:39.564516 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 27 01:10:39.564516 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application next(self.gen) Oct 27 01:10:39.564516 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 670, in _session Oct 27 01:10:39.564516 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application self.session.rollback() Oct 27 01:10:39.564516 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 27 01:10:39.564516 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application self.force_reraise() Oct 27 01:10:39.564516 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 27 01:10:39.564516 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise self.value Oct 27 01:10:39.564516 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 667, in _session Oct 27 01:10:39.564516 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 27 01:10:39.564516 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 695, in _end_session_transaction Oct 27 01:10:39.564516 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application session.commit() Oct 27 01:10:39.564516 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 27 01:10:39.564516 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 27 01:10:39.565422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 27 01:10:39.565422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application self._prepare_impl() Oct 27 01:10:39.565422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 27 01:10:39.565422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application self.session.flush() Oct 27 01:10:39.565422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 27 01:10:39.565422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application self._flush(objects) Oct 27 01:10:39.565422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 27 01:10:39.565422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 27 01:10:39.565422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 27 01:10:39.565422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application compat.raise_( Oct 27 01:10:39.565422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 27 01:10:39.565422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception Oct 27 01:10:39.565422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 27 01:10:39.565422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application flush_context.execute() Oct 27 01:10:39.565422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 27 01:10:39.565422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rec.execute(self) Oct 27 01:10:39.565422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 27 01:10:39.566273 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 27 01:10:39.566273 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 27 01:10:39.566273 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 27 01:10:39.566273 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 27 01:10:39.566273 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 27 01:10:39.566273 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 27 01:10:39.566273 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 27 01:10:39.566273 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 27 01:10:39.566273 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 27 01:10:39.566273 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 27 01:10:39.566273 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 27 01:10:39.566273 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 27 01:10:39.566273 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 27 01:10:39.566273 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 27 01:10:39.566273 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 27 01:10:39.567032 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 27 01:10:39.567032 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception Oct 27 01:10:39.567032 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 27 01:10:39.567032 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 27 01:10:39.567032 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 27 01:10:39.567032 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 27 01:10:39.567032 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 27 01:10:39.567032 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application result = self._query(query) Oct 27 01:10:39.567032 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 27 01:10:39.567032 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application conn.query(q) Oct 27 01:10:39.567032 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 27 01:10:39.567032 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 27 01:10:39.567032 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 27 01:10:39.567032 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application result.read() Oct 27 01:10:39.567032 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 27 01:10:39.567032 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 27 01:10:39.567032 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 27 01:10:39.567984 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 27 01:10:39.567984 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 27 01:10:39.567984 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 27 01:10:39.567984 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 27 01:10:39.567984 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 27 01:10:39.567984 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 27 01:10:39.567984 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 27 01:10:39.567984 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [parameters: {'id': '58d5548b0a61437eb9976a8eb7fc1eed', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 27 01:10:39.567984 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 27 01:10:39.567984 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:10:39.567984 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 27 01:10:39.567984 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:10:39.567984 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:10:39.567984 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 27 01:10:39.567984 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 27 01:10:39.570397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 27 01:10:39.570397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 27 01:10:39.570397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name Default. Oct 27 01:10:39.570397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:10:39.570397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 27 01:10:39.570397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:10:39.570397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:10:39.570397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:10:39.570397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:10:39.570397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:10:39.570397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:10:39.570397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:10:39.570397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:10:39.570397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:10:39.570397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:10:39.570397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:10:39.570397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:10:39.571345 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 128, in post Oct 27 01:10:39.571345 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_domain( Oct 27 01:10:39.571345 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:10:39.571345 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:10:39.571345 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 765, in create_domain Oct 27 01:10:39.571345 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application is_domain_project = self.create_project( Oct 27 01:10:39.571345 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:10:39.571345 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:10:39.571345 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 27 01:10:39.571345 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 27 01:10:39.571345 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 27 01:10:39.571345 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:10:39.572067 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e77f7241-81bd-4d47-ae60-4ca87b63b196 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:39.572154 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 8/16] 10.222.0.42 () {60 vars in 1137 bytes} [Thu Oct 27 01:10:39 2022] POST /identity/v3/domains => generated 190 bytes in 168 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Oct 27 01:10:39.572154 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14ec615-3ac3-4bde-9397-66f8e123f5ad None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:39.661375 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e77f7241-81bd-4d47-ae60-4ca87b63b196 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:39.669987 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14ec615-3ac3-4bde-9397-66f8e123f5ad None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:39.694369 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb9e34c-d6d6-490b-9e22-e1775d2b4049 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:39.695225 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb9e34c-d6d6-490b-9e22-e1775d2b4049 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:39.695502 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb9e34c-d6d6-490b-9e22-e1775d2b4049 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:39.695770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb9e34c-d6d6-490b-9e22-e1775d2b4049 None admin] PATH_INFO: `/v3/users/fe01809d18f644f9bc72669fc872b53d` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:39.697780 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fb9e34c-d6d6-490b-9e22-e1775d2b4049 None admin] RBAC: Authorizing `identity:get_user(user_id=fe01809d18f644f9bc72669fc872b53d)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:39.699233 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fb9e34c-d6d6-490b-9e22-e1775d2b4049 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:39.700777 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 9/17] 10.222.0.42 () {56 vars in 1177 bytes} [Thu Oct 27 01:10:39 2022] GET /identity/v3/users/fe01809d18f644f9bc72669fc872b53d => generated 247 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:10:39.704256 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82966779-f35e-416a-92fb-01db71bf2f81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:39.704882 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82966779-f35e-416a-92fb-01db71bf2f81 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:39.705188 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82966779-f35e-416a-92fb-01db71bf2f81 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:39.705461 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82966779-f35e-416a-92fb-01db71bf2f81 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:39.706255 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb9e34c-d6d6-490b-9e22-e1775d2b4049 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:39.715939 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82966779-f35e-416a-92fb-01db71bf2f81 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:39.717428 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82966779-f35e-416a-92fb-01db71bf2f81 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:39.726819 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:10:39.736469 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-82966779-f35e-416a-92fb-01db71bf2f81 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:10:39.736469 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:10:39.736469 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:10:39.736469 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:10:39.736469 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:10:39.736469 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:10:39.736469 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:10:39.736469 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:10:39.736469 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:10:39.736469 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:10:39.736469 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:10:39.736469 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:10:39.736469 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:10:39.736469 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:10:39.736469 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:10:39.736469 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:10:39.737604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:10:39.737604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:10:39.737604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:10:39.737604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:10:39.737604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:10:39.737604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:10:39.737604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:10:39.737604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:10:39.737604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:10:39.737604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:10:39.737604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:10:39.737604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:10:39.737604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:10:39.737604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:10:39.737604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:10:39.737604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:10:39.737604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:10:39.738615 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:10:39.738615 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:10:39.738615 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:10:39.738615 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 9/18] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:10:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:10:39.743068 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:10:39.743547 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82966779-f35e-416a-92fb-01db71bf2f81 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:39.810020 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb9e34c-d6d6-490b-9e22-e1775d2b4049 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:39.842333 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82966779-f35e-416a-92fb-01db71bf2f81 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:39.845175 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f53ad5-a181-4e06-9771-289d7cf7305c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:39.845828 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f53ad5-a181-4e06-9771-289d7cf7305c None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:39.846109 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f53ad5-a181-4e06-9771-289d7cf7305c None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:39.846379 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f53ad5-a181-4e06-9771-289d7cf7305c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:39.847490 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f53ad5-a181-4e06-9771-289d7cf7305c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:39.848942 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f53ad5-a181-4e06-9771-289d7cf7305c None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:39.850823 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 10/19] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:10:39 2022] GET /identity/v3/domains/default => generated 199 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:10:39.856365 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f53ad5-a181-4e06-9771-289d7cf7305c None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:39.875209 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f2116c9-217b-4f9e-9ba9-22fca8281f87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:39.875828 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2116c9-217b-4f9e-9ba9-22fca8281f87 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:39.876107 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2116c9-217b-4f9e-9ba9-22fca8281f87 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:39.876377 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2116c9-217b-4f9e-9ba9-22fca8281f87 None admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:39.877023 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f2116c9-217b-4f9e-9ba9-22fca8281f87 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:10:39.877359 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f2116c9-217b-4f9e-9ba9-22fca8281f87 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:39.878742 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f2116c9-217b-4f9e-9ba9-22fca8281f87 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:39.895747 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 10/20] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:10:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:10:39.963344 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f53ad5-a181-4e06-9771-289d7cf7305c None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:39.964930 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:10:39.999535 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c37a83e-9cca-417c-a044-9f1ca5892028 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:40.000206 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c37a83e-9cca-417c-a044-9f1ca5892028 None admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:40.000484 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c37a83e-9cca-417c-a044-9f1ca5892028 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:40.000756 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c37a83e-9cca-417c-a044-9f1ca5892028 None admin] PATH_INFO: `/v3/domains/default/users/fe01809d18f644f9bc72669fc872b53d/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:40.002758 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c37a83e-9cca-417c-a044-9f1ca5892028 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=fe01809d18f644f9bc72669fc872b53d, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:40.004240 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c37a83e-9cca-417c-a044-9f1ca5892028 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:40.122700 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 11/21] 10.222.0.42 () {58 vars in 1361 bytes} [Thu Oct 27 01:10:39 2022] PUT /identity/v3/domains/default/users/fe01809d18f644f9bc72669fc872b53d/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 267 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:10:42.886182 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711d847c-152d-4d68-aeee-8c9489308332 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:42.886688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711d847c-152d-4d68-aeee-8c9489308332 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:42.886791 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711d847c-152d-4d68-aeee-8c9489308332 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:42.888039 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 11/22] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:10:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:10:42.898612 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a0a4b8-ad14-4fb5-884d-a4a075041b0d None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:42.899113 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a0a4b8-ad14-4fb5-884d-a4a075041b0d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:42.899191 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a0a4b8-ad14-4fb5-884d-a4a075041b0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:42.938190 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-73a0a4b8-ad14-4fb5-884d-a4a075041b0d None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:10:42.939726 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-73a0a4b8-ad14-4fb5-884d-a4a075041b0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:10:42.966211 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74dc24d9-509c-4fc0-ad57-1f87ac461fbc None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:42.966506 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74dc24d9-509c-4fc0-ad57-1f87ac461fbc None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:42.966757 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74dc24d9-509c-4fc0-ad57-1f87ac461fbc None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:42.968205 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 12/23] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:10:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 27 01:10:42.979553 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6ac8c2-eef2-412a-b65b-13133534874e None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:42.979840 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6ac8c2-eef2-412a-b65b-13133534874e None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:42.980105 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6ac8c2-eef2-412a-b65b-13133534874e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:43.003097 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-fb6ac8c2-eef2-412a-b65b-13133534874e None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:10:43.006174 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-fb6ac8c2-eef2-412a-b65b-13133534874e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:10:43.117258 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:10:43.127759 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 12/24] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:10:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 231 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:10:43.128860 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3b13da-e4ab-4550-b6d2-eed6db02b592 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:43.129180 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3b13da-e4ab-4550-b6d2-eed6db02b592 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:43.129506 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3b13da-e4ab-4550-b6d2-eed6db02b592 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:43.152952 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 13/25] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:10:43 2022] GET /identity => generated 270 bytes in 25 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:10:43.168283 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 13/26] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:10:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 363 bytes (3 switches on core 0) Oct 27 01:10:43.178231 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ccedd03-fa6c-4be5-b64b-41ae8114ce44 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:43.178521 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ccedd03-fa6c-4be5-b64b-41ae8114ce44 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:43.178801 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ccedd03-fa6c-4be5-b64b-41ae8114ce44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:43.214715 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-1ccedd03-fa6c-4be5-b64b-41ae8114ce44 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:10:43.220064 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-1ccedd03-fa6c-4be5-b64b-41ae8114ce44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:10:43.361557 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 14/27] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:10:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 27 01:10:43.419846 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b11e3f3-52dc-4983-8d88-b3fc0c6cc790 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:43.420145 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b11e3f3-52dc-4983-8d88-b3fc0c6cc790 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:43.420395 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b11e3f3-52dc-4983-8d88-b3fc0c6cc790 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:43.421926 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 14/28] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:10:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:10:43.429364 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318b7646-4f46-40e4-9637-2542bbd0cbf6 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:43.429657 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318b7646-4f46-40e4-9637-2542bbd0cbf6 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:43.429937 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318b7646-4f46-40e4-9637-2542bbd0cbf6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:43.454484 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-318b7646-4f46-40e4-9637-2542bbd0cbf6 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:10:43.455752 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-318b7646-4f46-40e4-9637-2542bbd0cbf6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:10:43.476980 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:10:43.546067 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 15/29] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:10:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:10:43.609861 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b11e3f3-52dc-4983-8d88-b3fc0c6cc790 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:43.649362 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318b7646-4f46-40e4-9637-2542bbd0cbf6 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:43.789287 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b11e3f3-52dc-4983-8d88-b3fc0c6cc790 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:43.839540 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88ffb33-8a9b-4ccd-9a18-da3658f1df79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:43.840222 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318b7646-4f46-40e4-9637-2542bbd0cbf6 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:43.840546 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88ffb33-8a9b-4ccd-9a18-da3658f1df79 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:43.841025 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88ffb33-8a9b-4ccd-9a18-da3658f1df79 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:43.841486 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88ffb33-8a9b-4ccd-9a18-da3658f1df79 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:43.857092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88ffb33-8a9b-4ccd-9a18-da3658f1df79 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:43.858761 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88ffb33-8a9b-4ccd-9a18-da3658f1df79 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:43.878373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-b88ffb33-8a9b-4ccd-9a18-da3658f1df79 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:10:43.878373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:10:43.878373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:10:43.878373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:10:43.878373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:10:43.878373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:10:43.878373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:10:43.878373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:10:43.878373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:10:43.878373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:10:43.878373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:10:43.878373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:10:43.878373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:10:43.878373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:10:43.878373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:10:43.878373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:10:43.879477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:10:43.879477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:10:43.879477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:10:43.879477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:10:43.879477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:10:43.879477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:10:43.879477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:10:43.879477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:10:43.879477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:10:43.879477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:10:43.879477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:10:43.879477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:10:43.879477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:10:43.879477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:10:43.879477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:10:43.879477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:10:43.879477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:10:43.880537 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:10:43.880537 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:10:43.880537 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:10:43.883163 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 15/30] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:10:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 274 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:10:43.884496 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4291853-df3a-40ba-aeed-4f326c8ba0e5 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:43.884913 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4291853-df3a-40ba-aeed-4f326c8ba0e5 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:43.885355 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4291853-df3a-40ba-aeed-4f326c8ba0e5 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:43.887016 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 16/31] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:10:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:10:43.888264 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4291853-df3a-40ba-aeed-4f326c8ba0e5 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:43.907147 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1bca694-9f40-49bf-aa8c-632f19918e9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:43.907789 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bca694-9f40-49bf-aa8c-632f19918e9e None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:43.908067 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bca694-9f40-49bf-aa8c-632f19918e9e None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:43.908791 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bca694-9f40-49bf-aa8c-632f19918e9e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:43.924295 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1bca694-9f40-49bf-aa8c-632f19918e9e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:10:43.936998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-c1bca694-9f40-49bf-aa8c-632f19918e9e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:10:43.936998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:10:43.936998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:10:43.936998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:10:43.936998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:10:43.936998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:10:43.936998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:10:43.936998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:10:43.936998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:10:43.936998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:10:43.936998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:10:43.936998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:10:43.936998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:10:43.936998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:10:43.936998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:10:43.936998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:10:43.938102 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:10:43.938102 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:10:43.938102 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:10:43.938102 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:10:43.938102 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:10:43.938102 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:10:43.938102 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:10:43.938102 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:10:43.938102 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:10:43.938102 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:10:43.938102 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:10:43.938102 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:10:43.938102 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:10:43.938102 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:10:43.938102 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:10:43.938102 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:10:43.938990 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:10:43.938990 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:10:43.938990 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:10:43.938990 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:10:43.938990 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:10:43.938990 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:10:43.938990 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:10:43.938990 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:10:43.938990 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:10:43.938990 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:10:43.938990 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:10:43.938990 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:10:43.941235 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 16/32] 10.222.0.42 () {56 vars in 1096 bytes} [Thu Oct 27 01:10:43 2022] GET /identity/v3/roles/admin => generated 83 bytes in 292 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 27 01:10:43.942104 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1bca694-9f40-49bf-aa8c-632f19918e9e None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:43.942870 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:10:43.963784 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:10:44.007594 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4291853-df3a-40ba-aeed-4f326c8ba0e5 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:44.044214 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c04c9ba-8ba9-4faa-a192-c5f229355460 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:44.044749 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c04c9ba-8ba9-4faa-a192-c5f229355460 None admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:44.045022 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c04c9ba-8ba9-4faa-a192-c5f229355460 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:44.045324 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c04c9ba-8ba9-4faa-a192-c5f229355460 None admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:44.045995 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c04c9ba-8ba9-4faa-a192-c5f229355460 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:44.047366 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c04c9ba-8ba9-4faa-a192-c5f229355460 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:44.055631 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1bca694-9f40-49bf-aa8c-632f19918e9e None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:44.090602 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c84b07a-7e3c-40bc-85da-1358fcb853e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:44.091212 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c84b07a-7e3c-40bc-85da-1358fcb853e3 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:44.091485 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c84b07a-7e3c-40bc-85da-1358fcb853e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:44.091751 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c84b07a-7e3c-40bc-85da-1358fcb853e3 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:44.092397 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c84b07a-7e3c-40bc-85da-1358fcb853e3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:10:44.092707 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c84b07a-7e3c-40bc-85da-1358fcb853e3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:44.094243 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c84b07a-7e3c-40bc-85da-1358fcb853e3 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:44.120156 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 17/33] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:10:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:10:44.120156 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90fe81b-9611-4f1f-a88e-ea5a911aa246 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:44.120156 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90fe81b-9611-4f1f-a88e-ea5a911aa246 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:44.120156 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90fe81b-9611-4f1f-a88e-ea5a911aa246 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:44.136340 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-f90fe81b-9611-4f1f-a88e-ea5a911aa246 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:10:44.137542 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f90fe81b-9611-4f1f-a88e-ea5a911aa246 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:10:44.173122 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 17/34] 10.222.0.42 () {60 vars in 1131 bytes} [Thu Oct 27 01:10:43 2022] POST /identity/v3/roles => generated 225 bytes in 282 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:10:44.173122 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c04c9ba-8ba9-4faa-a192-c5f229355460 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:44.237237 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 18/35] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:10:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:10:44.238594 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90fe81b-9611-4f1f-a88e-ea5a911aa246 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:44.246673 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:10:44.327640 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:10:44.331741 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c04c9ba-8ba9-4faa-a192-c5f229355460 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:44.373613 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ffa797-0e0a-4dd4-9ec5-e1377d9636b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:44.374552 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ffa797-0e0a-4dd4-9ec5-e1377d9636b6 None admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:44.374975 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ffa797-0e0a-4dd4-9ec5-e1377d9636b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:44.375388 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ffa797-0e0a-4dd4-9ec5-e1377d9636b6 None admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:44.376243 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73ffa797-0e0a-4dd4-9ec5-e1377d9636b6 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:44.378022 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73ffa797-0e0a-4dd4-9ec5-e1377d9636b6 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:44.398617 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 18/36] 10.222.0.42 () {60 vars in 1131 bytes} [Thu Oct 27 01:10:44 2022] POST /identity/v3/roles => generated 219 bytes in 228 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:10:44.405204 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ffa797-0e0a-4dd4-9ec5-e1377d9636b6 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:44.405540 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90fe81b-9611-4f1f-a88e-ea5a911aa246 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:44.456667 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e5d931-68c1-438a-8fa9-f68bcb25878e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:44.458279 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e5d931-68c1-438a-8fa9-f68bcb25878e None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:44.458911 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e5d931-68c1-438a-8fa9-f68bcb25878e None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:44.459390 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e5d931-68c1-438a-8fa9-f68bcb25878e None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:44.460353 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28e5d931-68c1-438a-8fa9-f68bcb25878e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:10:44.461134 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28e5d931-68c1-438a-8fa9-f68bcb25878e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:44.463779 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28e5d931-68c1-438a-8fa9-f68bcb25878e None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:44.478516 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 19/37] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:10:44 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:10:44.481001 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b706d0d1-bcc9-4a7a-b706-a2d0d0897ce8 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:44.481612 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b706d0d1-bcc9-4a7a-b706-a2d0d0897ce8 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:44.482109 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b706d0d1-bcc9-4a7a-b706-a2d0d0897ce8 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:44.484188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 20/38] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:10:44 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:10:44.490286 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b706d0d1-bcc9-4a7a-b706-a2d0d0897ce8 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:44.573618 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ffa797-0e0a-4dd4-9ec5-e1377d9636b6 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:44.576752 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:10:44.610474 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea908648-602b-49ad-a412-a76acefb1c89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:44.611029 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea908648-602b-49ad-a412-a76acefb1c89 None admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:44.611314 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea908648-602b-49ad-a412-a76acefb1c89 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:44.611588 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea908648-602b-49ad-a412-a76acefb1c89 None admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:44.612242 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea908648-602b-49ad-a412-a76acefb1c89 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:44.613825 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea908648-602b-49ad-a412-a76acefb1c89 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:44.623308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b706d0d1-bcc9-4a7a-b706-a2d0d0897ce8 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:44.646949 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 19/39] 10.222.0.42 () {60 vars in 1141 bytes} [Thu Oct 27 01:10:44 2022] POST /identity/v3/projects => generated 301 bytes in 245 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:10:44.648461 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea93cda5-1ebe-4133-8059-c3f9ae92101f None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:44.648753 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea93cda5-1ebe-4133-8059-c3f9ae92101f None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:44.649038 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea93cda5-1ebe-4133-8059-c3f9ae92101f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:44.678613 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-ea93cda5-1ebe-4133-8059-c3f9ae92101f None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:10:44.680192 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-ea93cda5-1ebe-4133-8059-c3f9ae92101f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:10:44.689133 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:10:44.690509 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679e3e2b-2a3f-419e-9249-8fa0f2b85ea5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:44.691938 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679e3e2b-2a3f-419e-9249-8fa0f2b85ea5 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:44.692480 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679e3e2b-2a3f-419e-9249-8fa0f2b85ea5 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:44.693684 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679e3e2b-2a3f-419e-9249-8fa0f2b85ea5 None admin] PATH_INFO: `/v3/users/fe01809d18f644f9bc72669fc872b53d` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:44.696325 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679e3e2b-2a3f-419e-9249-8fa0f2b85ea5 None admin] RBAC: Authorizing `identity:get_user(user_id=fe01809d18f644f9bc72669fc872b53d)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:44.698718 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679e3e2b-2a3f-419e-9249-8fa0f2b85ea5 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:44.700556 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 21/40] 10.222.0.42 () {56 vars in 1177 bytes} [Thu Oct 27 01:10:44 2022] GET /identity/v3/users/fe01809d18f644f9bc72669fc872b53d => generated 247 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:10:44.701587 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679e3e2b-2a3f-419e-9249-8fa0f2b85ea5 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:44.764677 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 20/41] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:10:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:10:44.765592 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea93cda5-1ebe-4133-8059-c3f9ae92101f None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:44.834591 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679e3e2b-2a3f-419e-9249-8fa0f2b85ea5 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:44.876765 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:10:44.886093 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96cad049-c66e-4598-b4ef-cf961bfaadba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:44.886746 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96cad049-c66e-4598-b4ef-cf961bfaadba None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:44.887031 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96cad049-c66e-4598-b4ef-cf961bfaadba None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:44.887309 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96cad049-c66e-4598-b4ef-cf961bfaadba None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:44.887984 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96cad049-c66e-4598-b4ef-cf961bfaadba None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:44.891058 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96cad049-c66e-4598-b4ef-cf961bfaadba None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:44.897300 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea93cda5-1ebe-4133-8059-c3f9ae92101f None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:44.910937 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 22/42] 10.222.0.42 () {60 vars in 1131 bytes} [Thu Oct 27 01:10:44 2022] POST /identity/v3/roles => generated 223 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:10:44.941793 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57095ee-2ff4-4b32-8099-734db4726568 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:44.943091 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57095ee-2ff4-4b32-8099-734db4726568 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:44.943592 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57095ee-2ff4-4b32-8099-734db4726568 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:44.944062 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57095ee-2ff4-4b32-8099-734db4726568 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:44.945623 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57095ee-2ff4-4b32-8099-734db4726568 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:44.947122 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57095ee-2ff4-4b32-8099-734db4726568 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:44.949035 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 21/43] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:10:44 2022] GET /identity/v3/domains/default => generated 199 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:10:44.955131 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96cad049-c66e-4598-b4ef-cf961bfaadba None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:45.052763 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57095ee-2ff4-4b32-8099-734db4726568 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:45.065794 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:10:45.066425 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96cad049-c66e-4598-b4ef-cf961bfaadba None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:45.103618 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9166e348-99ff-414d-bb5f-6a70ae3666fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:45.104272 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9166e348-99ff-414d-bb5f-6a70ae3666fe None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:45.104560 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9166e348-99ff-414d-bb5f-6a70ae3666fe None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:45.104839 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9166e348-99ff-414d-bb5f-6a70ae3666fe None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:45.105638 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9166e348-99ff-414d-bb5f-6a70ae3666fe None admin] RBAC: Adding query filter params (role.id=414286927fdd44688107ff5e10235acf, scope.domain.id=default, user.id=fe01809d18f644f9bc72669fc872b53d) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:10:45.105955 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9166e348-99ff-414d-bb5f-6a70ae3666fe None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:45.107462 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9166e348-99ff-414d-bb5f-6a70ae3666fe None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:45.119378 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 23/44] 10.222.0.42 () {56 vars in 1322 bytes} [Thu Oct 27 01:10:44 2022] GET /identity/v3/role_assignments?user.id=fe01809d18f644f9bc72669fc872b53d&scope.domain.id=default&role.id=414286927fdd44688107ff5e10235acf => generated 530 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:10:45.274156 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57095ee-2ff4-4b32-8099-734db4726568 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:45.308732 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49fca4a4-d302-4b54-9ea6-3e3326e39e6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:45.309396 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fca4a4-d302-4b54-9ea6-3e3326e39e6a None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:45.309677 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fca4a4-d302-4b54-9ea6-3e3326e39e6a None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:45.309957 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fca4a4-d302-4b54-9ea6-3e3326e39e6a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:45.311046 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49fca4a4-d302-4b54-9ea6-3e3326e39e6a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:45.312479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49fca4a4-d302-4b54-9ea6-3e3326e39e6a None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:45.314083 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 22/45] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:10:45 2022] GET /identity/v3/domains/default => generated 199 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:10:45.320384 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9166e348-99ff-414d-bb5f-6a70ae3666fe None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:45.377185 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:10:45.421229 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9166e348-99ff-414d-bb5f-6a70ae3666fe None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:45.454215 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9133c765-9a51-4b2a-a869-82e21908fac1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:45.454820 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9133c765-9a51-4b2a-a869-82e21908fac1 None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:45.455105 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9133c765-9a51-4b2a-a869-82e21908fac1 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:45.455381 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9133c765-9a51-4b2a-a869-82e21908fac1 None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:45.456044 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9133c765-9a51-4b2a-a869-82e21908fac1 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:45.457525 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9133c765-9a51-4b2a-a869-82e21908fac1 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:45.488954 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 24/46] 10.222.0.42 () {60 vars in 1141 bytes} [Thu Oct 27 01:10:45 2022] POST /identity/v3/projects => generated 312 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:10:47.632021 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649274c3-98ac-4a52-83e5-af420a50910a None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:47.632543 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649274c3-98ac-4a52-83e5-af420a50910a None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:47.632543 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649274c3-98ac-4a52-83e5-af420a50910a None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:47.633939 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 23/47] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:10:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:10:47.639131 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d53124-bf14-4061-8d66-7c8a1e0fc649 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:47.639490 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d53124-bf14-4061-8d66-7c8a1e0fc649 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:47.639731 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d53124-bf14-4061-8d66-7c8a1e0fc649 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:47.658343 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-b8d53124-bf14-4061-8d66-7c8a1e0fc649 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:10:47.659536 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-b8d53124-bf14-4061-8d66-7c8a1e0fc649 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:10:47.724820 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 25/48] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:10:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:10:47.998308 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649274c3-98ac-4a52-83e5-af420a50910a None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:48.076610 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:10:48.099024 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649274c3-98ac-4a52-83e5-af420a50910a None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:48.131258 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858d2ff8-91fa-404a-b10e-b2834adcd726 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:48.131881 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858d2ff8-91fa-404a-b10e-b2834adcd726 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:48.132160 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858d2ff8-91fa-404a-b10e-b2834adcd726 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:48.132436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858d2ff8-91fa-404a-b10e-b2834adcd726 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:48.133567 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-858d2ff8-91fa-404a-b10e-b2834adcd726 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:48.135007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-858d2ff8-91fa-404a-b10e-b2834adcd726 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:48.136558 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 24/49] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:10:47 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:10:48.142194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d53124-bf14-4061-8d66-7c8a1e0fc649 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:48.180045 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:10:48.238037 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d53124-bf14-4061-8d66-7c8a1e0fc649 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:48.270381 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8297d4-aff9-4fe2-8ccf-b13e33cc14c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:48.271519 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8297d4-aff9-4fe2-8ccf-b13e33cc14c7 None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:48.271876 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8297d4-aff9-4fe2-8ccf-b13e33cc14c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:48.272150 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8297d4-aff9-4fe2-8ccf-b13e33cc14c7 None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:48.272804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e8297d4-aff9-4fe2-8ccf-b13e33cc14c7 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:48.274273 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e8297d4-aff9-4fe2-8ccf-b13e33cc14c7 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:48.298232 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 26/50] 10.222.0.42 () {60 vars in 1140 bytes} [Thu Oct 27 01:10:48 2022] POST /identity/v3/projects => generated 298 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:10:50.459189 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35e8d2c-d7c2-4b8b-b093-37e6185c5d71 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:50.459707 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35e8d2c-d7c2-4b8b-b093-37e6185c5d71 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:50.459822 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35e8d2c-d7c2-4b8b-b093-37e6185c5d71 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:50.461105 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 25/51] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:10:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:10:50.466316 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a2e147-7937-4cdb-bb7c-8be0d4f7b080 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:50.466710 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a2e147-7937-4cdb-bb7c-8be0d4f7b080 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:50.466872 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a2e147-7937-4cdb-bb7c-8be0d4f7b080 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:50.486087 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-b7a2e147-7937-4cdb-bb7c-8be0d4f7b080 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:10:50.487267 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-b7a2e147-7937-4cdb-bb7c-8be0d4f7b080 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:10:50.655445 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 27/52] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:10:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:10:50.932360 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35e8d2c-d7c2-4b8b-b093-37e6185c5d71 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:51.031117 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35e8d2c-d7c2-4b8b-b093-37e6185c5d71 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:51.063852 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4d4c12-9c2b-447a-a4fe-308933dabc30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:51.064413 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4d4c12-9c2b-447a-a4fe-308933dabc30 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:51.064693 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4d4c12-9c2b-447a-a4fe-308933dabc30 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:51.064967 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4d4c12-9c2b-447a-a4fe-308933dabc30 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:51.066101 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c4d4c12-9c2b-447a-a4fe-308933dabc30 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:51.067537 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c4d4c12-9c2b-447a-a4fe-308933dabc30 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:51.069614 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 26/53] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:10:50 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:10:51.075206 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a2e147-7937-4cdb-bb7c-8be0d4f7b080 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:51.096567 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:10:51.172225 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a2e147-7937-4cdb-bb7c-8be0d4f7b080 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:51.204602 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be9873d6-836b-4b6e-ad51-db3111687b3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:51.205252 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9873d6-836b-4b6e-ad51-db3111687b3e None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:51.205534 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9873d6-836b-4b6e-ad51-db3111687b3e None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:51.205805 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9873d6-836b-4b6e-ad51-db3111687b3e None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:51.206463 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be9873d6-836b-4b6e-ad51-db3111687b3e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:51.207899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be9873d6-836b-4b6e-ad51-db3111687b3e None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:51.235937 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 28/54] 10.222.0.42 () {60 vars in 1132 bytes} [Thu Oct 27 01:10:51 2022] POST /identity/v3/users => generated 275 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:10:54.515929 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed5b825-e512-43d4-b0b7-a288490281ee None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:54.516893 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed5b825-e512-43d4-b0b7-a288490281ee None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:54.517429 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed5b825-e512-43d4-b0b7-a288490281ee None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:54.519234 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 27/55] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:10:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:10:54.534672 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6285d859-99a6-44bb-b496-91dd20a55c3f None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:54.535551 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6285d859-99a6-44bb-b496-91dd20a55c3f None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:54.536040 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6285d859-99a6-44bb-b496-91dd20a55c3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:54.566546 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-6285d859-99a6-44bb-b496-91dd20a55c3f None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:10:54.568750 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-6285d859-99a6-44bb-b496-91dd20a55c3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:10:54.668794 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 29/56] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:10:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 27 01:10:54.751573 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f9c6fa-fffc-40c7-923d-d2d18159e395 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:54.752209 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f9c6fa-fffc-40c7-923d-d2d18159e395 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:54.752671 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f9c6fa-fffc-40c7-923d-d2d18159e395 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:54.754251 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 28/57] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:10:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:10:54.761761 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819e1db9-8e98-4292-92ba-fde81b45edec None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:54.762291 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819e1db9-8e98-4292-92ba-fde81b45edec None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:54.762836 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819e1db9-8e98-4292-92ba-fde81b45edec None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:54.764247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 29/58] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:10:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 27 01:10:54.765152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ba3b88-69ac-4f1a-afa8-a1f41876f08e None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:54.765742 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ba3b88-69ac-4f1a-afa8-a1f41876f08e None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:54.766268 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ba3b88-69ac-4f1a-afa8-a1f41876f08e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:54.768996 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3059277-37e9-4fe9-93a0-d36334ce0602 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:54.769508 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3059277-37e9-4fe9-93a0-d36334ce0602 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:54.769997 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3059277-37e9-4fe9-93a0-d36334ce0602 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:54.797093 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:10:54.800884 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-84ba3b88-69ac-4f1a-afa8-a1f41876f08e None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:10:54.802250 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-d3059277-37e9-4fe9-93a0-d36334ce0602 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:10:54.803464 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-d3059277-37e9-4fe9-93a0-d36334ce0602 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:10:54.806849 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-84ba3b88-69ac-4f1a-afa8-a1f41876f08e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:10:54.825197 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:10:54.937711 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 30/59] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:10:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:10:54.938211 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 30/60] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:10:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 27 01:10:55.070312 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ba3b88-69ac-4f1a-afa8-a1f41876f08e None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:55.136252 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e08a70e-2899-40a7-a442-63ecddb12ace None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:55.136764 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e08a70e-2899-40a7-a442-63ecddb12ace None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:55.137213 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e08a70e-2899-40a7-a442-63ecddb12ace None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:55.138718 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 31/61] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:10:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:10:55.144707 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67126ecf-915a-418f-ac1e-f1272ce330c1 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:55.145175 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67126ecf-915a-418f-ac1e-f1272ce330c1 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:55.145595 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67126ecf-915a-418f-ac1e-f1272ce330c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:55.178647 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-67126ecf-915a-418f-ac1e-f1272ce330c1 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:10:55.180231 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-67126ecf-915a-418f-ac1e-f1272ce330c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:10:55.228857 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ba3b88-69ac-4f1a-afa8-a1f41876f08e None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:55.269835 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f12c9f-b7fd-4471-a7b2-541462cc7837 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:55.270588 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f12c9f-b7fd-4471-a7b2-541462cc7837 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:55.271506 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f12c9f-b7fd-4471-a7b2-541462cc7837 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:55.272661 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f12c9f-b7fd-4471-a7b2-541462cc7837 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:55.275661 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:10:55.281328 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 32/62] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:10:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 27 01:10:55.282238 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67126ecf-915a-418f-ac1e-f1272ce330c1 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:55.284390 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f12c9f-b7fd-4471-a7b2-541462cc7837 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:10:55.293196 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-b0f12c9f-b7fd-4471-a7b2-541462cc7837 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 27 01:10:55.293196 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:10:55.293196 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:10:55.293196 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:10:55.293196 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:10:55.293196 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:10:55.293196 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:10:55.293196 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:10:55.293196 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:10:55.293196 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:10:55.293196 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:10:55.293196 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:10:55.293196 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:10:55.293196 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:10:55.293196 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:10:55.293196 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:10:55.294088 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:10:55.294088 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:10:55.294088 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:10:55.294088 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:10:55.294088 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:10:55.294088 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:10:55.294088 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:10:55.294088 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:10:55.294088 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:10:55.294088 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:10:55.294088 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:10:55.294088 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:10:55.294088 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:10:55.294088 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:10:55.294088 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:10:55.294088 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:10:55.294862 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:10:55.294862 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:10:55.294862 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:10:55.294862 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:10:55.294862 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:10:55.294862 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:10:55.294862 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:10:55.294862 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:10:55.294862 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:10:55.294862 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:10:55.294862 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 27 01:10:55.294862 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:10:55.294862 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 31/63] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:10:55 2022] GET /identity/v3/roles/member => generated 84 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:10:55.294862 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f12c9f-b7fd-4471-a7b2-541462cc7837 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:55.371823 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:10:55.397819 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67126ecf-915a-418f-ac1e-f1272ce330c1 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:55.410280 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f12c9f-b7fd-4471-a7b2-541462cc7837 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:55.433127 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c535b3-fbef-4257-883a-42ca821d4591 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:55.433765 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c535b3-fbef-4257-883a-42ca821d4591 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:55.434057 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c535b3-fbef-4257-883a-42ca821d4591 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:55.434324 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c535b3-fbef-4257-883a-42ca821d4591 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:55.444734 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c535b3-fbef-4257-883a-42ca821d4591 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:10:55.448182 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714f2c49-fd1a-45d4-ae72-c166a54d2fa2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:55.449001 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714f2c49-fd1a-45d4-ae72-c166a54d2fa2 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:55.449507 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714f2c49-fd1a-45d4-ae72-c166a54d2fa2 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:55.449619 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-c2c535b3-fbef-4257-883a-42ca821d4591 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 27 01:10:55.449619 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:10:55.449619 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:10:55.449619 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:10:55.449619 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:10:55.449619 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:10:55.449619 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:10:55.449619 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:10:55.449619 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:10:55.449619 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:10:55.449619 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:10:55.449619 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:10:55.449619 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:10:55.449619 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:10:55.449619 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:10:55.449619 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:10:55.450394 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:10:55.450394 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:10:55.450394 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:10:55.450394 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:10:55.450394 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:10:55.450394 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:10:55.450394 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:10:55.450394 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:10:55.450394 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:10:55.450394 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:10:55.450394 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:10:55.450394 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:10:55.450394 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:10:55.450394 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:10:55.450394 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:10:55.450394 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:10:55.451144 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:10:55.451144 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:10:55.451144 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:10:55.451144 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:10:55.451144 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:10:55.451144 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:10:55.451144 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:10:55.451144 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:10:55.451144 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:10:55.451144 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:10:55.451144 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 27 01:10:55.451144 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:10:55.451144 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 33/64] 10.222.0.42 () {56 vars in 1114 bytes} [Thu Oct 27 01:10:55 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:10:55.452395 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714f2c49-fd1a-45d4-ae72-c166a54d2fa2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:55.452553 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c535b3-fbef-4257-883a-42ca821d4591 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:55.465009 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-714f2c49-fd1a-45d4-ae72-c166a54d2fa2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:10:55.470561 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-714f2c49-fd1a-45d4-ae72-c166a54d2fa2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:10:55.470561 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:10:55.470561 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:10:55.470561 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:10:55.470561 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:10:55.470561 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:10:55.470561 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:10:55.470561 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:10:55.470561 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:10:55.470561 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:10:55.470561 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:10:55.470561 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:10:55.470561 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:10:55.470561 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:10:55.470561 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:10:55.470561 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:10:55.471391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:10:55.471391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:10:55.471391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:10:55.471391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:10:55.471391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:10:55.471391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:10:55.471391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:10:55.471391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:10:55.471391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:10:55.471391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:10:55.471391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:10:55.471391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:10:55.471391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:10:55.471391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:10:55.471391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:10:55.471391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:10:55.472135 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:10:55.472135 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:10:55.472135 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:10:55.472135 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:10:55.472135 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:10:55.472135 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:10:55.472135 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:10:55.472135 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:10:55.472135 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:10:55.472135 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:10:55.472135 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:10:55.472135 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:10:55.474534 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 32/65] 10.222.0.42 () {56 vars in 1096 bytes} [Thu Oct 27 01:10:55 2022] GET /identity/v3/roles/admin => generated 83 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:10:55.475819 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3784e6bc-54f2-43df-b356-dbd3e37ad087 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:55.476242 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3784e6bc-54f2-43df-b356-dbd3e37ad087 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:55.477151 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3784e6bc-54f2-43df-b356-dbd3e37ad087 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:55.478598 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 33/66] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:10:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:10:55.479426 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3784e6bc-54f2-43df-b356-dbd3e37ad087 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:55.541785 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:10:55.561165 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c535b3-fbef-4257-883a-42ca821d4591 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:55.591563 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3784e6bc-54f2-43df-b356-dbd3e37ad087 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:55.596130 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033aea7f-96e9-4e9f-8b69-a702e2c9921e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:55.596734 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033aea7f-96e9-4e9f-8b69-a702e2c9921e None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:55.597011 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033aea7f-96e9-4e9f-8b69-a702e2c9921e None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:55.597314 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033aea7f-96e9-4e9f-8b69-a702e2c9921e None admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:55.597978 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033aea7f-96e9-4e9f-8b69-a702e2c9921e None admin] RBAC: Adding query filter params (name=member) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:10:55.598274 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033aea7f-96e9-4e9f-8b69-a702e2c9921e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:55.599808 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033aea7f-96e9-4e9f-8b69-a702e2c9921e None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:55.601358 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:10:55.612764 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 34/67] 10.222.0.42 () {56 vars in 1101 bytes} [Thu Oct 27 01:10:55 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:10:55.613435 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033aea7f-96e9-4e9f-8b69-a702e2c9921e None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:55.630736 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f4ad62-3c20-43a7-b3d4-603e48c77910 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:55.631596 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f4ad62-3c20-43a7-b3d4-603e48c77910 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:55.631875 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f4ad62-3c20-43a7-b3d4-603e48c77910 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:55.632365 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f4ad62-3c20-43a7-b3d4-603e48c77910 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:55.632810 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f4ad62-3c20-43a7-b3d4-603e48c77910 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:10:55.633145 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f4ad62-3c20-43a7-b3d4-603e48c77910 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:55.634536 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f4ad62-3c20-43a7-b3d4-603e48c77910 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:55.642400 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 34/68] 10.222.0.42 () {56 vars in 1111 bytes} [Thu Oct 27 01:10:55 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:10:55.643530 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6893196a-fb31-4cc0-9c34-8e8442a950a3 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:55.643806 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6893196a-fb31-4cc0-9c34-8e8442a950a3 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:55.644089 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6893196a-fb31-4cc0-9c34-8e8442a950a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:55.661979 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-6893196a-fb31-4cc0-9c34-8e8442a950a3 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:10:55.663174 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-6893196a-fb31-4cc0-9c34-8e8442a950a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:10:55.711382 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033aea7f-96e9-4e9f-8b69-a702e2c9921e None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:55.733924 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 35/69] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:10:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:10:55.734586 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6893196a-fb31-4cc0-9c34-8e8442a950a3 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:55.744679 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320babb9-c96c-4111-b491-3a15c90af684 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:55.745346 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320babb9-c96c-4111-b491-3a15c90af684 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:55.745626 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320babb9-c96c-4111-b491-3a15c90af684 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:55.745891 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320babb9-c96c-4111-b491-3a15c90af684 None admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:55.746559 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-320babb9-c96c-4111-b491-3a15c90af684 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:10:55.746866 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-320babb9-c96c-4111-b491-3a15c90af684 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:55.748228 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-320babb9-c96c-4111-b491-3a15c90af684 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:55.758766 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 35/70] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:10:55 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:10:55.759657 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320babb9-c96c-4111-b491-3a15c90af684 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:55.794513 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:10:55.828074 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:10:55.854769 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6893196a-fb31-4cc0-9c34-8e8442a950a3 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:55.873273 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320babb9-c96c-4111-b491-3a15c90af684 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:55.892523 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c030a1d5-0ab2-4985-adab-5032b1a9c985 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:55.893106 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c030a1d5-0ab2-4985-adab-5032b1a9c985 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:55.893394 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c030a1d5-0ab2-4985-adab-5032b1a9c985 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:55.893668 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c030a1d5-0ab2-4985-adab-5032b1a9c985 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:55.905012 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c030a1d5-0ab2-4985-adab-5032b1a9c985 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:10:55.909766 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7599e35-098e-4b2e-9423-520a6cf147a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:55.909888 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-c030a1d5-0ab2-4985-adab-5032b1a9c985 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 27 01:10:55.909888 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:10:55.909888 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:10:55.909888 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:10:55.909888 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:10:55.909888 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:10:55.909888 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:10:55.909888 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:10:55.909888 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:10:55.909888 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:10:55.909888 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:10:55.909888 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:10:55.909888 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:10:55.909888 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:10:55.909888 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:10:55.909888 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:10:55.910694 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:10:55.910694 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:10:55.910694 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:10:55.910694 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:10:55.910694 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:10:55.910694 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:10:55.910694 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:10:55.910694 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:10:55.910694 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:10:55.910694 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:10:55.910694 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:10:55.910694 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:10:55.910694 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:10:55.910694 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:10:55.910694 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:10:55.910694 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:10:55.911712 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:10:55.911712 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:10:55.911712 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:10:55.911712 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:10:55.911712 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:10:55.911712 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:10:55.911712 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:10:55.911712 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:10:55.911712 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:10:55.911712 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:10:55.911712 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 27 01:10:55.911712 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:10:55.911712 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 36/71] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:10:55 2022] GET /identity/v3/roles/member => generated 84 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:10:55.912341 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c030a1d5-0ab2-4985-adab-5032b1a9c985 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:55.913312 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7599e35-098e-4b2e-9423-520a6cf147a1 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:55.913753 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7599e35-098e-4b2e-9423-520a6cf147a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:55.914168 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7599e35-098e-4b2e-9423-520a6cf147a1 None admin] PATH_INFO: `/v3/users/4ed8fe9600cf453685d7c281db64fb87` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:55.929289 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7599e35-098e-4b2e-9423-520a6cf147a1 None admin] RBAC: Authorizing `identity:get_user(user_id=4ed8fe9600cf453685d7c281db64fb87)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:55.930735 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7599e35-098e-4b2e-9423-520a6cf147a1 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:55.932927 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 36/72] 10.222.0.42 () {56 vars in 1177 bytes} [Thu Oct 27 01:10:55 2022] GET /identity/v3/users/4ed8fe9600cf453685d7c281db64fb87 => generated 275 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:10:55.933726 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7599e35-098e-4b2e-9423-520a6cf147a1 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:56.025936 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c030a1d5-0ab2-4985-adab-5032b1a9c985 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:56.035289 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:10:56.037800 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7599e35-098e-4b2e-9423-520a6cf147a1 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:56.054208 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:10:56.061554 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9da58723-a18e-48e4-b337-cf41ab0b23a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:56.062180 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da58723-a18e-48e4-b337-cf41ab0b23a4 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:56.062455 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da58723-a18e-48e4-b337-cf41ab0b23a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:56.062727 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da58723-a18e-48e4-b337-cf41ab0b23a4 None admin] PATH_INFO: `/v3/users/4ed8fe9600cf453685d7c281db64fb87` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:56.063851 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9da58723-a18e-48e4-b337-cf41ab0b23a4 None admin] RBAC: Authorizing `identity:get_user(user_id=4ed8fe9600cf453685d7c281db64fb87)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:56.065317 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9da58723-a18e-48e4-b337-cf41ab0b23a4 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:56.066891 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 37/73] 10.222.0.42 () {56 vars in 1177 bytes} [Thu Oct 27 01:10:55 2022] GET /identity/v3/users/4ed8fe9600cf453685d7c281db64fb87 => generated 275 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:10:56.068215 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9da58723-a18e-48e4-b337-cf41ab0b23a4 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:56.077718 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129b1f0b-9555-40e8-b051-0ff24bf4d132 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:56.078360 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129b1f0b-9555-40e8-b051-0ff24bf4d132 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:56.078635 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129b1f0b-9555-40e8-b051-0ff24bf4d132 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:56.078903 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129b1f0b-9555-40e8-b051-0ff24bf4d132 None admin] PATH_INFO: `/v3/users/fe01809d18f644f9bc72669fc872b53d` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:56.080052 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129b1f0b-9555-40e8-b051-0ff24bf4d132 None admin] RBAC: Authorizing `identity:get_user(user_id=fe01809d18f644f9bc72669fc872b53d)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:56.081534 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129b1f0b-9555-40e8-b051-0ff24bf4d132 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:56.083099 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 37/74] 10.222.0.42 () {56 vars in 1177 bytes} [Thu Oct 27 01:10:55 2022] GET /identity/v3/users/fe01809d18f644f9bc72669fc872b53d => generated 247 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:10:56.083927 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129b1f0b-9555-40e8-b051-0ff24bf4d132 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:56.165043 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9da58723-a18e-48e4-b337-cf41ab0b23a4 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:56.182734 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129b1f0b-9555-40e8-b051-0ff24bf4d132 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:56.198941 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be94ee1-83f7-4cf3-b845-b453118a1e06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:56.199546 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be94ee1-83f7-4cf3-b845-b453118a1e06 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:56.199823 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be94ee1-83f7-4cf3-b845-b453118a1e06 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:56.200088 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be94ee1-83f7-4cf3-b845-b453118a1e06 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:56.200749 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be94ee1-83f7-4cf3-b845-b453118a1e06 None admin] RBAC: Adding query filter params (name=member) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:10:56.201046 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be94ee1-83f7-4cf3-b845-b453118a1e06 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:56.202427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be94ee1-83f7-4cf3-b845-b453118a1e06 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:56.210861 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 38/75] 10.222.0.42 () {56 vars in 1101 bytes} [Thu Oct 27 01:10:56 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:10:56.211729 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be94ee1-83f7-4cf3-b845-b453118a1e06 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:56.222782 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb85036e-3fca-4478-8521-46ff08987c97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:56.223412 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb85036e-3fca-4478-8521-46ff08987c97 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:56.223689 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb85036e-3fca-4478-8521-46ff08987c97 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:56.224028 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb85036e-3fca-4478-8521-46ff08987c97 None admin] PATH_INFO: `/v3/projects/50a39f634ddf41798003dd395f14be68` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:56.225193 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb85036e-3fca-4478-8521-46ff08987c97 None admin] RBAC: Authorizing `identity:get_project(project_id=50a39f634ddf41798003dd395f14be68)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:56.226639 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb85036e-3fca-4478-8521-46ff08987c97 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:56.228253 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 38/76] 10.222.0.42 () {56 vars in 1186 bytes} [Thu Oct 27 01:10:56 2022] GET /identity/v3/projects/50a39f634ddf41798003dd395f14be68 => generated 298 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:10:56.229008 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb85036e-3fca-4478-8521-46ff08987c97 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:56.249696 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:10:56.282804 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:10:56.311945 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be94ee1-83f7-4cf3-b845-b453118a1e06 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:56.331598 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb85036e-3fca-4478-8521-46ff08987c97 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:56.343943 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f800df1-e89e-4994-a428-04f38ade9d5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:56.344568 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f800df1-e89e-4994-a428-04f38ade9d5e None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:56.344845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f800df1-e89e-4994-a428-04f38ade9d5e None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:56.345143 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f800df1-e89e-4994-a428-04f38ade9d5e None admin] PATH_INFO: `/v3/domains/default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:56.346230 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f800df1-e89e-4994-a428-04f38ade9d5e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:56.347665 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f800df1-e89e-4994-a428-04f38ade9d5e None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:56.350274 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 39/77] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:10:56 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:10:56.352290 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f800df1-e89e-4994-a428-04f38ade9d5e None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:56.365475 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5601793-627d-4db3-9de2-c9a1c83a9a2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:56.366109 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5601793-627d-4db3-9de2-c9a1c83a9a2b None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:56.366390 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5601793-627d-4db3-9de2-c9a1c83a9a2b None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:56.366666 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5601793-627d-4db3-9de2-c9a1c83a9a2b None admin] PATH_INFO: `/v3/projects/50a39f634ddf41798003dd395f14be68` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:56.367791 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5601793-627d-4db3-9de2-c9a1c83a9a2b None admin] RBAC: Authorizing `identity:get_project(project_id=50a39f634ddf41798003dd395f14be68)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:56.369356 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5601793-627d-4db3-9de2-c9a1c83a9a2b None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:56.370971 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 39/78] 10.222.0.42 () {56 vars in 1186 bytes} [Thu Oct 27 01:10:56 2022] GET /identity/v3/projects/50a39f634ddf41798003dd395f14be68 => generated 298 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:10:56.371728 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5601793-627d-4db3-9de2-c9a1c83a9a2b None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:56.456384 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f800df1-e89e-4994-a428-04f38ade9d5e None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:56.463233 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:10:56.471405 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5601793-627d-4db3-9de2-c9a1c83a9a2b None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:56.491078 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ae9a88-a4d5-42c3-ada3-6b738cc7e157 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:56.491728 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ae9a88-a4d5-42c3-ada3-6b738cc7e157 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:56.492007 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ae9a88-a4d5-42c3-ada3-6b738cc7e157 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:56.492117 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:10:56.492281 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ae9a88-a4d5-42c3-ada3-6b738cc7e157 None admin] PATH_INFO: `/v3/projects/50a39f634ddf41798003dd395f14be68` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:56.493485 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00ae9a88-a4d5-42c3-ada3-6b738cc7e157 None admin] RBAC: Authorizing `identity:get_project(project_id=50a39f634ddf41798003dd395f14be68)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:56.495130 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00ae9a88-a4d5-42c3-ada3-6b738cc7e157 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:56.496749 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 40/79] 10.222.0.42 () {56 vars in 1186 bytes} [Thu Oct 27 01:10:56 2022] GET /identity/v3/projects/50a39f634ddf41798003dd395f14be68 => generated 298 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:10:56.497504 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ae9a88-a4d5-42c3-ada3-6b738cc7e157 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:56.506677 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61fcae1-df20-42ba-97aa-1b9d85d77d21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:56.507310 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61fcae1-df20-42ba-97aa-1b9d85d77d21 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:56.507590 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61fcae1-df20-42ba-97aa-1b9d85d77d21 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:56.507867 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61fcae1-df20-42ba-97aa-1b9d85d77d21 None admin] PATH_INFO: `/v3/users/4ed8fe9600cf453685d7c281db64fb87` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:56.509005 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61fcae1-df20-42ba-97aa-1b9d85d77d21 None admin] RBAC: Authorizing `identity:get_user(user_id=4ed8fe9600cf453685d7c281db64fb87)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:56.510464 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61fcae1-df20-42ba-97aa-1b9d85d77d21 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:56.512159 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 40/80] 10.222.0.42 () {56 vars in 1177 bytes} [Thu Oct 27 01:10:56 2022] GET /identity/v3/users/4ed8fe9600cf453685d7c281db64fb87 => generated 275 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:10:56.513821 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61fcae1-df20-42ba-97aa-1b9d85d77d21 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:56.596566 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ae9a88-a4d5-42c3-ada3-6b738cc7e157 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:56.610677 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61fcae1-df20-42ba-97aa-1b9d85d77d21 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:56.630764 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc46c6d-8970-432d-b337-ef1214040f56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:56.631525 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc46c6d-8970-432d-b337-ef1214040f56 None admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:56.631807 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc46c6d-8970-432d-b337-ef1214040f56 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:56.632083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc46c6d-8970-432d-b337-ef1214040f56 None admin] PATH_INFO: `/v3/projects/50a39f634ddf41798003dd395f14be68/users/4ed8fe9600cf453685d7c281db64fb87/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:56.634036 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc46c6d-8970-432d-b337-ef1214040f56 None admin] RBAC: Authorizing `identity:create_grant(project_id=50a39f634ddf41798003dd395f14be68, user_id=4ed8fe9600cf453685d7c281db64fb87, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:56.635491 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc46c6d-8970-432d-b337-ef1214040f56 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:56.644909 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a34c6b8-7e64-4566-a376-8189685534d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:56.645555 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a34c6b8-7e64-4566-a376-8189685534d3 None admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:56.645864 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a34c6b8-7e64-4566-a376-8189685534d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:56.646143 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a34c6b8-7e64-4566-a376-8189685534d3 None admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:56.646797 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a34c6b8-7e64-4566-a376-8189685534d3 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:56.647538 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 41/81] 10.222.0.42 () {58 vars in 1439 bytes} [Thu Oct 27 01:10:56 2022] PUT /identity/v3/projects/50a39f634ddf41798003dd395f14be68/users/4ed8fe9600cf453685d7c281db64fb87/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:10:56.648230 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a34c6b8-7e64-4566-a376-8189685534d3 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:56.648328 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc46c6d-8970-432d-b337-ef1214040f56 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:56.685719 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:10:56.694558 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 41/82] 10.222.0.42 () {60 vars in 1132 bytes} [Thu Oct 27 01:10:56 2022] POST /identity/v3/users => generated 289 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:10:56.695607 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a34c6b8-7e64-4566-a376-8189685534d3 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:56.750905 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:10:56.772317 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc46c6d-8970-432d-b337-ef1214040f56 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:56.823188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c30c228-fc3e-4e76-99b4-79d896982b25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:56.824356 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c30c228-fc3e-4e76-99b4-79d896982b25 None admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:56.824643 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c30c228-fc3e-4e76-99b4-79d896982b25 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:56.824923 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c30c228-fc3e-4e76-99b4-79d896982b25 None admin] PATH_INFO: `/v3/projects/50a39f634ddf41798003dd395f14be68/users/4ed8fe9600cf453685d7c281db64fb87/roles/bb9bd8ef24ff4aa5b639a2bd8567345b` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:56.827686 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c30c228-fc3e-4e76-99b4-79d896982b25 None admin] RBAC: Authorizing `identity:create_grant(project_id=50a39f634ddf41798003dd395f14be68, user_id=4ed8fe9600cf453685d7c281db64fb87, role_id=bb9bd8ef24ff4aa5b639a2bd8567345b)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:56.829304 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c30c228-fc3e-4e76-99b4-79d896982b25 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:56.831868 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a34c6b8-7e64-4566-a376-8189685534d3 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:56.849604 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 42/83] 10.222.0.42 () {58 vars in 1439 bytes} [Thu Oct 27 01:10:56 2022] PUT /identity/v3/projects/50a39f634ddf41798003dd395f14be68/users/4ed8fe9600cf453685d7c281db64fb87/roles/bb9bd8ef24ff4aa5b639a2bd8567345b => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:10:56.854167 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c30c228-fc3e-4e76-99b4-79d896982b25 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:56.915443 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0667ce2-ec3e-4670-8bc9-0ba44b4ca50e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:56.916434 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0667ce2-ec3e-4670-8bc9-0ba44b4ca50e None admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:56.916936 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0667ce2-ec3e-4670-8bc9-0ba44b4ca50e None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:56.917371 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0667ce2-ec3e-4670-8bc9-0ba44b4ca50e None admin] PATH_INFO: `/v3/projects/50a39f634ddf41798003dd395f14be68/users/fe01809d18f644f9bc72669fc872b53d/roles/414286927fdd44688107ff5e10235acf` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:56.919725 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0667ce2-ec3e-4670-8bc9-0ba44b4ca50e None admin] RBAC: Authorizing `identity:create_grant(project_id=50a39f634ddf41798003dd395f14be68, user_id=fe01809d18f644f9bc72669fc872b53d, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:56.922026 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0667ce2-ec3e-4670-8bc9-0ba44b4ca50e None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:56.946386 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 42/84] 10.222.0.42 () {58 vars in 1439 bytes} [Thu Oct 27 01:10:56 2022] PUT /identity/v3/projects/50a39f634ddf41798003dd395f14be68/users/fe01809d18f644f9bc72669fc872b53d/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:10:57.011761 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c30c228-fc3e-4e76-99b4-79d896982b25 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:57.014840 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:10:57.057468 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-004e9228-46c7-476e-864c-bb0eb002bbd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:57.058127 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004e9228-46c7-476e-864c-bb0eb002bbd2 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:57.058408 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004e9228-46c7-476e-864c-bb0eb002bbd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:57.058743 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004e9228-46c7-476e-864c-bb0eb002bbd2 None admin] PATH_INFO: `/v3/projects/d7d874be296c401193685c18ced01979` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:57.060019 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-004e9228-46c7-476e-864c-bb0eb002bbd2 None admin] RBAC: Authorizing `identity:get_project(project_id=d7d874be296c401193685c18ced01979)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:57.061606 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-004e9228-46c7-476e-864c-bb0eb002bbd2 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:57.063383 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 43/85] 10.222.0.42 () {56 vars in 1186 bytes} [Thu Oct 27 01:10:56 2022] GET /identity/v3/projects/d7d874be296c401193685c18ced01979 => generated 312 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:10:57.070910 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0667ce2-ec3e-4670-8bc9-0ba44b4ca50e None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:10:57.191753 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0667ce2-ec3e-4670-8bc9-0ba44b4ca50e None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:10:57.194858 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:10:57.232845 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643a8205-3531-4395-a1e1-0749980f83ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:10:57.233638 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643a8205-3531-4395-a1e1-0749980f83ca None admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:57.233919 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643a8205-3531-4395-a1e1-0749980f83ca None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:57.234188 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643a8205-3531-4395-a1e1-0749980f83ca None admin] PATH_INFO: `/v3/projects/d7d874be296c401193685c18ced01979/users/4ed8fe9600cf453685d7c281db64fb87/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:57.236165 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-643a8205-3531-4395-a1e1-0749980f83ca None admin] RBAC: Authorizing `identity:create_grant(project_id=d7d874be296c401193685c18ced01979, user_id=4ed8fe9600cf453685d7c281db64fb87, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:10:57.237640 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-643a8205-3531-4395-a1e1-0749980f83ca None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:10:57.254592 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 43/86] 10.222.0.42 () {58 vars in 1439 bytes} [Thu Oct 27 01:10:57 2022] PUT /identity/v3/projects/d7d874be296c401193685c18ced01979/users/4ed8fe9600cf453685d7c281db64fb87/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:10:59.483168 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c070e2-85c2-4f11-9291-a39277354c8d None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:59.484210 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c070e2-85c2-4f11-9291-a39277354c8d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:59.484683 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c070e2-85c2-4f11-9291-a39277354c8d None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:59.486551 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 44/87] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:10:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:10:59.492523 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d528f8e2-7fc9-49c5-8888-2ac533fdc4fe None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:59.493363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d528f8e2-7fc9-49c5-8888-2ac533fdc4fe None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:59.493884 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d528f8e2-7fc9-49c5-8888-2ac533fdc4fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:59.519611 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-d528f8e2-7fc9-49c5-8888-2ac533fdc4fe None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:10:59.521286 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-d528f8e2-7fc9-49c5-8888-2ac533fdc4fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:10:59.549974 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e197788-2d2e-4897-b5b1-c2268c6fb9b1 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:59.550572 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e197788-2d2e-4897-b5b1-c2268c6fb9b1 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:59.551035 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e197788-2d2e-4897-b5b1-c2268c6fb9b1 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:59.553354 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 45/88] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:10:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 27 01:10:59.561027 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78ee419-ab2e-4e60-8acb-559bb0199599 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:59.561980 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78ee419-ab2e-4e60-8acb-559bb0199599 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:59.562746 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78ee419-ab2e-4e60-8acb-559bb0199599 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:59.593684 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-b78ee419-ab2e-4e60-8acb-559bb0199599 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:10:59.595919 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-b78ee419-ab2e-4e60-8acb-559bb0199599 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:10:59.625144 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 44/89] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:10:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:10:59.626794 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93b0d92-51ca-4339-af8b-8dc9ea96b98d None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:59.627480 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93b0d92-51ca-4339-af8b-8dc9ea96b98d None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:59.629388 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93b0d92-51ca-4339-af8b-8dc9ea96b98d None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:59.631708 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 45/90] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:10:59 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:10:59.637330 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7582f7c-522f-42fe-be32-69c8d6dbacb2 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:10:59.638876 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7582f7c-522f-42fe-be32-69c8d6dbacb2 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:10:59.640580 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7582f7c-522f-42fe-be32-69c8d6dbacb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:10:59.684328 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-b7582f7c-522f-42fe-be32-69c8d6dbacb2 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:10:59.686700 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-b7582f7c-522f-42fe-be32-69c8d6dbacb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:10:59.726141 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 46/91] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:10:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 27 01:10:59.782665 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 46/92] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:10:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:11:00.044744 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78ee419-ab2e-4e60-8acb-559bb0199599 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:00.084053 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7582f7c-522f-42fe-be32-69c8d6dbacb2 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:00.122419 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:00.200763 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:00.218258 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78ee419-ab2e-4e60-8acb-559bb0199599 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:00.270060 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7582f7c-522f-42fe-be32-69c8d6dbacb2 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:00.289787 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e7e310-d9f2-48f2-87e9-275219ce7369 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:00.290442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e7e310-d9f2-48f2-87e9-275219ce7369 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:00.290725 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e7e310-d9f2-48f2-87e9-275219ce7369 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:00.291430 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e7e310-d9f2-48f2-87e9-275219ce7369 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:00.313404 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e7e310-d9f2-48f2-87e9-275219ce7369 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:00.318770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-e0e7e310-d9f2-48f2-87e9-275219ce7369 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 27 01:11:00.318770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:00.318770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:00.318770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:00.318770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:00.318770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:00.318770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:00.318770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:00.318770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:00.318770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:00.318770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:00.318770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:00.318770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:00.318770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:00.318770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:00.318770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:00.319657 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:00.319657 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:00.319657 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:00.319657 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:00.319657 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:00.319657 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:00.319657 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:00.319657 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:00.319657 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:00.319657 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:00.319657 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:00.319657 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:00.319657 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:00.319657 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:00.319657 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:00.319657 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:00.320412 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:00.320412 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:00.320412 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:00.320412 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:00.320412 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:00.320412 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:00.320412 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:00.320412 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:00.320412 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:00.320412 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:00.320412 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 27 01:11:00.320412 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:00.320412 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 47/93] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:11:00 2022] GET /identity/v3/roles/member => generated 84 bytes in 276 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:00.321025 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e7e310-d9f2-48f2-87e9-275219ce7369 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:00.340676 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32abbce-17a5-4271-ac53-29d4070ad8b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:00.341363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32abbce-17a5-4271-ac53-29d4070ad8b5 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:00.341647 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32abbce-17a5-4271-ac53-29d4070ad8b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:00.341922 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32abbce-17a5-4271-ac53-29d4070ad8b5 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:00.366609 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d32abbce-17a5-4271-ac53-29d4070ad8b5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:00.374167 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-d32abbce-17a5-4271-ac53-29d4070ad8b5 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 27 01:11:00.374167 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:00.374167 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:00.374167 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:00.374167 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:00.374167 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:00.374167 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:00.374167 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:00.374167 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:00.374167 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:00.374167 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:00.374167 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:00.374167 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:00.374167 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:00.374167 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:00.374167 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:00.375007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:00.375007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:00.375007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:00.375007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:00.375007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:00.375007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:00.375007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:00.375007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:00.375007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:00.375007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:00.375007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:00.375007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:00.375007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:00.375007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:00.375007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:00.375007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:00.375782 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:00.375782 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:00.375782 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:00.375782 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:00.375782 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:00.375782 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:00.375782 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:00.375782 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:00.375782 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:00.375782 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:00.375782 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 27 01:11:00.375782 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:00.382495 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 47/94] 10.222.0.42 () {56 vars in 1114 bytes} [Thu Oct 27 01:11:00 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 299 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 27 01:11:00.386269 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32abbce-17a5-4271-ac53-29d4070ad8b5 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:00.516058 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e7e310-d9f2-48f2-87e9-275219ce7369 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:00.535774 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:00.566279 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32abbce-17a5-4271-ac53-29d4070ad8b5 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:00.589858 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d88527d-360b-4d28-a8c8-e3f70f4cd69b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:00.590509 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d88527d-360b-4d28-a8c8-e3f70f4cd69b None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:00.590790 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d88527d-360b-4d28-a8c8-e3f70f4cd69b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:00.591063 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d88527d-360b-4d28-a8c8-e3f70f4cd69b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:00.614322 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d88527d-360b-4d28-a8c8-e3f70f4cd69b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:00.620336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-8d88527d-360b-4d28-a8c8-e3f70f4cd69b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:11:00.620336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:00.620336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:00.620336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:00.620336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:00.620336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:00.620336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:00.620336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:00.620336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:00.620336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:00.620336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:00.620336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:00.620336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:00.620336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:00.620336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:00.620336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:00.621300 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:00.621300 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:00.621300 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:00.621300 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:00.621300 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:00.621300 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:00.621300 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:00.621300 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:00.621300 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:00.621300 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:00.621300 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:00.621300 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:00.621300 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:00.621300 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:00.621300 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:00.621300 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:00.622079 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:00.622079 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:00.622079 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:00.622079 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:00.622079 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:00.622079 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:00.622079 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:00.622079 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:00.622079 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:00.622079 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:00.622079 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:11:00.622079 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:00.622654 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 48/95] 10.222.0.42 () {56 vars in 1096 bytes} [Thu Oct 27 01:11:00 2022] GET /identity/v3/roles/admin => generated 83 bytes in 302 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:00.623472 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d88527d-360b-4d28-a8c8-e3f70f4cd69b None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:00.644804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:00.653096 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25b41fb-5861-4713-a604-adcbb9bbac10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:00.653722 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25b41fb-5861-4713-a604-adcbb9bbac10 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:00.653998 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25b41fb-5861-4713-a604-adcbb9bbac10 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:00.654266 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25b41fb-5861-4713-a604-adcbb9bbac10 None admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:00.654933 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d25b41fb-5861-4713-a604-adcbb9bbac10 None admin] RBAC: Adding query filter params (name=member) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:00.655229 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d25b41fb-5861-4713-a604-adcbb9bbac10 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:00.656730 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d25b41fb-5861-4713-a604-adcbb9bbac10 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:00.677875 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 48/96] 10.222.0.42 () {56 vars in 1101 bytes} [Thu Oct 27 01:11:00 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:00.679443 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25b41fb-5861-4713-a604-adcbb9bbac10 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:00.746850 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d88527d-360b-4d28-a8c8-e3f70f4cd69b None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:00.792212 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94e977d-3d66-4f91-96a5-28eccc9056e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:00.792841 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94e977d-3d66-4f91-96a5-28eccc9056e6 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:00.793162 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94e977d-3d66-4f91-96a5-28eccc9056e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:00.793438 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94e977d-3d66-4f91-96a5-28eccc9056e6 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:00.794119 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a94e977d-3d66-4f91-96a5-28eccc9056e6 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:00.794423 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a94e977d-3d66-4f91-96a5-28eccc9056e6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:00.795827 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a94e977d-3d66-4f91-96a5-28eccc9056e6 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:00.804837 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 49/97] 10.222.0.42 () {56 vars in 1111 bytes} [Thu Oct 27 01:11:00 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:00.805564 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94e977d-3d66-4f91-96a5-28eccc9056e6 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:00.832088 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25b41fb-5861-4713-a604-adcbb9bbac10 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:00.868552 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02f04a0-c2b0-40cf-b67f-77e74cefd0e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:00.869185 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02f04a0-c2b0-40cf-b67f-77e74cefd0e2 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:00.869470 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02f04a0-c2b0-40cf-b67f-77e74cefd0e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:00.869744 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02f04a0-c2b0-40cf-b67f-77e74cefd0e2 None admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:00.870416 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02f04a0-c2b0-40cf-b67f-77e74cefd0e2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:00.870720 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02f04a0-c2b0-40cf-b67f-77e74cefd0e2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:00.872113 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02f04a0-c2b0-40cf-b67f-77e74cefd0e2 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:00.881747 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 49/98] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:11:00 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:00.882899 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d649f89f-edf0-4198-9ea8-875bbeb6a8da None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:00.883183 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d649f89f-edf0-4198-9ea8-875bbeb6a8da None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:00.883483 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d649f89f-edf0-4198-9ea8-875bbeb6a8da None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:00.884648 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 50/99] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:00 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:00.885442 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d649f89f-edf0-4198-9ea8-875bbeb6a8da None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:00.893914 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:00.896786 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:00.920247 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94e977d-3d66-4f91-96a5-28eccc9056e6 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:00.956531 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a993ad-f9aa-4578-ab4e-a87bc96aab0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:00.957193 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a993ad-f9aa-4578-ab4e-a87bc96aab0b None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:00.957478 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a993ad-f9aa-4578-ab4e-a87bc96aab0b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:00.957755 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a993ad-f9aa-4578-ab4e-a87bc96aab0b None admin] PATH_INFO: `/v3/users/4ed8fe9600cf453685d7c281db64fb87` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:00.959543 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a993ad-f9aa-4578-ab4e-a87bc96aab0b None admin] RBAC: Authorizing `identity:get_user(user_id=4ed8fe9600cf453685d7c281db64fb87)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:00.961005 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a993ad-f9aa-4578-ab4e-a87bc96aab0b None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:00.962628 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 50/100] 10.222.0.42 () {56 vars in 1177 bytes} [Thu Oct 27 01:11:00 2022] GET /identity/v3/users/4ed8fe9600cf453685d7c281db64fb87 => generated 275 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:00.963755 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f59ec2-c594-48a5-9c44-3e0a18795302 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:00.964041 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f59ec2-c594-48a5-9c44-3e0a18795302 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:00.964320 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f59ec2-c594-48a5-9c44-3e0a18795302 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:00.965556 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 51/101] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:00 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:00.966572 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8a6e89-8722-4566-895a-cbd41ca4eb74 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:00.966869 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8a6e89-8722-4566-895a-cbd41ca4eb74 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:00.967203 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8a6e89-8722-4566-895a-cbd41ca4eb74 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:00.968410 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 52/102] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:00 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:00.972878 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8a6e89-8722-4566-895a-cbd41ca4eb74 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:00.993485 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d649f89f-edf0-4198-9ea8-875bbeb6a8da None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:01.026806 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e2197bf-c024-4958-ba59-2f64b7967624 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:01.027430 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2197bf-c024-4958-ba59-2f64b7967624 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:01.027713 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2197bf-c024-4958-ba59-2f64b7967624 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:01.027983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2197bf-c024-4958-ba59-2f64b7967624 None admin] PATH_INFO: `/v3/users/4ed8fe9600cf453685d7c281db64fb87` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:01.029127 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e2197bf-c024-4958-ba59-2f64b7967624 None admin] RBAC: Authorizing `identity:get_user(user_id=4ed8fe9600cf453685d7c281db64fb87)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:01.030611 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e2197bf-c024-4958-ba59-2f64b7967624 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:01.032327 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 51/103] 10.222.0.42 () {56 vars in 1177 bytes} [Thu Oct 27 01:11:00 2022] GET /identity/v3/users/4ed8fe9600cf453685d7c281db64fb87 => generated 275 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:01.033281 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eaf1af2-3d80-482e-b4d9-a6d01331098d None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:01.033560 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eaf1af2-3d80-482e-b4d9-a6d01331098d None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:01.033830 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eaf1af2-3d80-482e-b4d9-a6d01331098d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:01.051253 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-9eaf1af2-3d80-482e-b4d9-a6d01331098d None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:01.052932 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-9eaf1af2-3d80-482e-b4d9-a6d01331098d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:01.074347 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8a6e89-8722-4566-895a-cbd41ca4eb74 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:01.107421 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730297c8-a2a5-4b8e-bce0-4f98247b2fa7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:01.107968 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730297c8-a2a5-4b8e-bce0-4f98247b2fa7 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:01.108246 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730297c8-a2a5-4b8e-bce0-4f98247b2fa7 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:01.108521 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730297c8-a2a5-4b8e-bce0-4f98247b2fa7 None admin] PATH_INFO: `/v3/users/fe01809d18f644f9bc72669fc872b53d` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:01.109638 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-730297c8-a2a5-4b8e-bce0-4f98247b2fa7 None admin] RBAC: Authorizing `identity:get_user(user_id=fe01809d18f644f9bc72669fc872b53d)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:01.111066 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-730297c8-a2a5-4b8e-bce0-4f98247b2fa7 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:01.112624 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 53/104] 10.222.0.42 () {56 vars in 1177 bytes} [Thu Oct 27 01:11:00 2022] GET /identity/v3/users/fe01809d18f644f9bc72669fc872b53d => generated 247 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:01.113406 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730297c8-a2a5-4b8e-bce0-4f98247b2fa7 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:01.119378 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 52/105] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:11:01.127139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e4f799-2b57-4d00-aef9-ed9c230e1e65 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:01.127709 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:01.127813 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e4f799-2b57-4d00-aef9-ed9c230e1e65 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:01.128009 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e4f799-2b57-4d00-aef9-ed9c230e1e65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:01.143708 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:01.151526 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-c9e4f799-2b57-4d00-aef9-ed9c230e1e65 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:01.152908 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-c9e4f799-2b57-4d00-aef9-ed9c230e1e65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:01.221722 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730297c8-a2a5-4b8e-bce0-4f98247b2fa7 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:01.237320 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 53/106] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:11:01.237320 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acee2c1d-a5b1-47c2-9960-2ac89c68a88c None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:01.237320 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acee2c1d-a5b1-47c2-9960-2ac89c68a88c None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:01.237320 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acee2c1d-a5b1-47c2-9960-2ac89c68a88c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:01.257422 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b5be327-9389-4361-9847-c6656ed9d4e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:01.257925 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-acee2c1d-a5b1-47c2-9960-2ac89c68a88c None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:01.258042 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5be327-9389-4361-9847-c6656ed9d4e9 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:01.258312 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5be327-9389-4361-9847-c6656ed9d4e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:01.258591 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5be327-9389-4361-9847-c6656ed9d4e9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:01.259447 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-acee2c1d-a5b1-47c2-9960-2ac89c68a88c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:01.337865 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 54/107] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:11:01.338833 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acee2c1d-a5b1-47c2-9960-2ac89c68a88c None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:01.342633 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b5be327-9389-4361-9847-c6656ed9d4e9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:01.344114 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b5be327-9389-4361-9847-c6656ed9d4e9 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:01.357213 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:01.409307 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:01.449147 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acee2c1d-a5b1-47c2-9960-2ac89c68a88c None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:01.471864 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 54/108] 10.222.0.42 () {58 vars in 1282 bytes} [Thu Oct 27 01:11:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 27 01:11:01.472894 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b5be327-9389-4361-9847-c6656ed9d4e9 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:01.499354 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c269a449-3480-49a8-bdbe-ba5307646bbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:01.500215 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c269a449-3480-49a8-bdbe-ba5307646bbb None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:01.500676 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c269a449-3480-49a8-bdbe-ba5307646bbb None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:01.501223 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c269a449-3480-49a8-bdbe-ba5307646bbb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:01.577318 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c269a449-3480-49a8-bdbe-ba5307646bbb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:01.579147 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c269a449-3480-49a8-bdbe-ba5307646bbb None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:01.584859 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b5be327-9389-4361-9847-c6656ed9d4e9 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:01.596905 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:01.624663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5ce5db-6867-4a5f-8066-b405f09728d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:01.625309 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5ce5db-6867-4a5f-8066-b405f09728d6 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:01.625705 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5ce5db-6867-4a5f-8066-b405f09728d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:01.625986 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5ce5db-6867-4a5f-8066-b405f09728d6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:01.644444 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:01.683024 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 55/109] 10.222.0.42 () {58 vars in 1282 bytes} [Thu Oct 27 01:11:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 27 01:11:01.683024 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c269a449-3480-49a8-bdbe-ba5307646bbb None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:01.696303 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c5ce5db-6867-4a5f-8066-b405f09728d6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:01.697812 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c5ce5db-6867-4a5f-8066-b405f09728d6 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:01.782001 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c269a449-3480-49a8-bdbe-ba5307646bbb None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:01.794131 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 55/110] 10.222.0.42 () {58 vars in 1282 bytes} [Thu Oct 27 01:11:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 27 01:11:01.794966 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5ce5db-6867-4a5f-8066-b405f09728d6 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:01.806116 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:01.823883 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58762bff-f215-483f-95a3-4359d9ddb4d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:01.824608 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58762bff-f215-483f-95a3-4359d9ddb4d3 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:01.824957 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58762bff-f215-483f-95a3-4359d9ddb4d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:01.825280 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58762bff-f215-483f-95a3-4359d9ddb4d3 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:01.835305 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58762bff-f215-483f-95a3-4359d9ddb4d3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:01.840254 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-58762bff-f215-483f-95a3-4359d9ddb4d3 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 27 01:11:01.840254 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:01.840254 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:01.840254 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:01.840254 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:01.840254 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:01.840254 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:01.840254 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:01.840254 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:01.840254 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:01.840254 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:01.840254 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:01.840254 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:01.840254 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:01.840254 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:01.840254 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:01.841292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:01.841292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:01.841292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:01.841292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:01.841292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:01.841292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:01.841292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:01.841292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:01.841292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:01.841292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:01.841292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:01.841292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:01.841292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:01.841292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:01.841292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:01.841292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:01.842053 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:01.842053 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:01.842053 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:01.842053 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:01.842053 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:01.842053 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:01.842053 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:01.842053 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:01.842053 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:01.842053 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:01.842053 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 27 01:11:01.842053 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:01.842053 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 56/111] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:11:01 2022] GET /identity/v3/roles/reader => generated 84 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:01.842880 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58762bff-f215-483f-95a3-4359d9ddb4d3 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:01.869806 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:01.900561 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5ce5db-6867-4a5f-8066-b405f09728d6 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:01.934191 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316d20d3-8d13-4040-8621-413b63e056b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:01.934746 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316d20d3-8d13-4040-8621-413b63e056b0 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:01.935027 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316d20d3-8d13-4040-8621-413b63e056b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:01.935306 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316d20d3-8d13-4040-8621-413b63e056b0 None admin] PATH_INFO: `/v3/projects/50a39f634ddf41798003dd395f14be68` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:01.936426 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-316d20d3-8d13-4040-8621-413b63e056b0 None admin] RBAC: Authorizing `identity:get_project(project_id=50a39f634ddf41798003dd395f14be68)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:01.937918 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-316d20d3-8d13-4040-8621-413b63e056b0 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:01.939542 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 56/112] 10.222.0.42 () {56 vars in 1186 bytes} [Thu Oct 27 01:11:01 2022] GET /identity/v3/projects/50a39f634ddf41798003dd395f14be68 => generated 298 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:01.940296 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316d20d3-8d13-4040-8621-413b63e056b0 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:01.940723 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58762bff-f215-483f-95a3-4359d9ddb4d3 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:01.976768 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b179c0-4368-4265-a68e-449a65996b42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:01.977439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b179c0-4368-4265-a68e-449a65996b42 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:01.977716 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b179c0-4368-4265-a68e-449a65996b42 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:01.977986 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b179c0-4368-4265-a68e-449a65996b42 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:01.987663 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26b179c0-4368-4265-a68e-449a65996b42 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:01.993128 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-26b179c0-4368-4265-a68e-449a65996b42 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 27 01:11:01.993128 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:01.993128 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:01.993128 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:01.993128 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:01.993128 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:01.993128 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:01.993128 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:01.993128 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:01.993128 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:01.993128 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:01.993128 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:01.993128 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:01.993128 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:01.993128 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:01.993128 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:01.994168 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:01.994168 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:01.994168 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:01.994168 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:01.994168 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:01.994168 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:01.994168 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:01.994168 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:01.994168 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:01.994168 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:01.994168 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:01.994168 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:01.994168 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:01.994168 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:01.994168 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:01.994168 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:01.994940 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:01.994940 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:01.994940 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:01.994940 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:01.994940 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:01.994940 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:01.994940 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:01.994940 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:01.994940 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:01.994940 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:01.994940 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 27 01:11:01.994940 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:01.994940 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 57/113] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:11:01 2022] GET /identity/v3/roles/member => generated 84 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:01.995564 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b179c0-4368-4265-a68e-449a65996b42 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:02.020952 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:02.042014 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316d20d3-8d13-4040-8621-413b63e056b0 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:02.075276 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ce0b97-d942-47e4-85b2-3e8cd22405c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:02.075896 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ce0b97-d942-47e4-85b2-3e8cd22405c1 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:02.076172 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ce0b97-d942-47e4-85b2-3e8cd22405c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:02.076445 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ce0b97-d942-47e4-85b2-3e8cd22405c1 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:02.077421 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:02.077587 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ce0b97-d942-47e4-85b2-3e8cd22405c1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:02.079134 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ce0b97-d942-47e4-85b2-3e8cd22405c1 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:02.080786 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 57/114] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:11:01 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:02.085906 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ce0b97-d942-47e4-85b2-3e8cd22405c1 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:02.092918 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b179c0-4368-4265-a68e-449a65996b42 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:02.126517 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb3dd1b-6c8f-4761-9758-4dfb471c1044 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:02.127149 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb3dd1b-6c8f-4761-9758-4dfb471c1044 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:02.127429 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb3dd1b-6c8f-4761-9758-4dfb471c1044 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:02.127704 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb3dd1b-6c8f-4761-9758-4dfb471c1044 None admin] PATH_INFO: `/v3/projects/50a39f634ddf41798003dd395f14be68` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:02.128840 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcb3dd1b-6c8f-4761-9758-4dfb471c1044 None admin] RBAC: Authorizing `identity:get_project(project_id=50a39f634ddf41798003dd395f14be68)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:02.130895 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcb3dd1b-6c8f-4761-9758-4dfb471c1044 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:02.132525 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 58/115] 10.222.0.42 () {56 vars in 1186 bytes} [Thu Oct 27 01:11:01 2022] GET /identity/v3/projects/50a39f634ddf41798003dd395f14be68 => generated 298 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:02.137943 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb3dd1b-6c8f-4761-9758-4dfb471c1044 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:02.185518 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ce0b97-d942-47e4-85b2-3e8cd22405c1 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:02.219126 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f891f4-b798-44ce-a545-a64603bb4ce9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:02.219748 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f891f4-b798-44ce-a545-a64603bb4ce9 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:02.220028 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f891f4-b798-44ce-a545-a64603bb4ce9 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:02.220330 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f891f4-b798-44ce-a545-a64603bb4ce9 None admin] PATH_INFO: `/v3/projects/50a39f634ddf41798003dd395f14be68` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:02.221479 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95f891f4-b798-44ce-a545-a64603bb4ce9 None admin] RBAC: Authorizing `identity:get_project(project_id=50a39f634ddf41798003dd395f14be68)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:02.222908 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95f891f4-b798-44ce-a545-a64603bb4ce9 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:02.225104 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 58/116] 10.222.0.42 () {56 vars in 1186 bytes} [Thu Oct 27 01:11:02 2022] GET /identity/v3/projects/50a39f634ddf41798003dd395f14be68 => generated 298 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:02.225870 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f891f4-b798-44ce-a545-a64603bb4ce9 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:02.234005 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb3dd1b-6c8f-4761-9758-4dfb471c1044 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:02.241243 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:02.267928 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb85131-feaa-4b08-bd37-fe0b18ba8762 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:02.268545 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb85131-feaa-4b08-bd37-fe0b18ba8762 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:02.268827 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb85131-feaa-4b08-bd37-fe0b18ba8762 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:02.269135 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb85131-feaa-4b08-bd37-fe0b18ba8762 None admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:02.269806 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb85131-feaa-4b08-bd37-fe0b18ba8762 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:02.270113 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb85131-feaa-4b08-bd37-fe0b18ba8762 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:02.271557 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb85131-feaa-4b08-bd37-fe0b18ba8762 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:02.279437 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 59/117] 10.222.0.42 () {56 vars in 1101 bytes} [Thu Oct 27 01:11:02 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:02.280220 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb85131-feaa-4b08-bd37-fe0b18ba8762 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:02.298478 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:02.329213 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f891f4-b798-44ce-a545-a64603bb4ce9 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:02.364851 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc554fbc-bbec-4823-af67-cc3b08f2075e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:02.365490 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc554fbc-bbec-4823-af67-cc3b08f2075e None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:02.365773 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc554fbc-bbec-4823-af67-cc3b08f2075e None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:02.366046 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc554fbc-bbec-4823-af67-cc3b08f2075e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:02.366774 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc554fbc-bbec-4823-af67-cc3b08f2075e None admin] RBAC: Adding query filter params (role.id=f199b79a7178462fa67bb8243943408f, scope.project.id=50a39f634ddf41798003dd395f14be68, user.id=4ed8fe9600cf453685d7c281db64fb87) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:02.367082 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc554fbc-bbec-4823-af67-cc3b08f2075e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:02.368531 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc554fbc-bbec-4823-af67-cc3b08f2075e None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:02.374365 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 59/118] 10.222.0.42 () {56 vars in 1374 bytes} [Thu Oct 27 01:11:02 2022] GET /identity/v3/role_assignments?user.id=4ed8fe9600cf453685d7c281db64fb87&scope.project.id=50a39f634ddf41798003dd395f14be68&role.id=f199b79a7178462fa67bb8243943408f => generated 608 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:02.375148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc554fbc-bbec-4823-af67-cc3b08f2075e None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:02.377955 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb85131-feaa-4b08-bd37-fe0b18ba8762 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:02.414894 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c69b720-e1a9-4fbb-83ca-ce1156bfcd07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:02.415521 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c69b720-e1a9-4fbb-83ca-ce1156bfcd07 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:02.415806 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c69b720-e1a9-4fbb-83ca-ce1156bfcd07 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:02.416080 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c69b720-e1a9-4fbb-83ca-ce1156bfcd07 None admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:02.416765 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c69b720-e1a9-4fbb-83ca-ce1156bfcd07 None admin] RBAC: Adding query filter params (name=member) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:02.417090 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c69b720-e1a9-4fbb-83ca-ce1156bfcd07 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:02.418504 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c69b720-e1a9-4fbb-83ca-ce1156bfcd07 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:02.427859 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 60/119] 10.222.0.42 () {56 vars in 1101 bytes} [Thu Oct 27 01:11:02 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:02.433243 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c69b720-e1a9-4fbb-83ca-ce1156bfcd07 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:02.474716 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:02.502026 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc554fbc-bbec-4823-af67-cc3b08f2075e None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:02.523162 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:02.541715 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e13a8cb-b751-4745-b780-f1db6ba5f2ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:02.542355 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e13a8cb-b751-4745-b780-f1db6ba5f2ff None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:02.542640 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e13a8cb-b751-4745-b780-f1db6ba5f2ff None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:02.542916 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e13a8cb-b751-4745-b780-f1db6ba5f2ff None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:02.543606 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e13a8cb-b751-4745-b780-f1db6ba5f2ff None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:02.545084 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e13a8cb-b751-4745-b780-f1db6ba5f2ff None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:02.550101 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c69b720-e1a9-4fbb-83ca-ce1156bfcd07 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:02.575423 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 60/120] 10.222.0.42 () {60 vars in 1141 bytes} [Thu Oct 27 01:11:02 2022] POST /identity/v3/projects => generated 302 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:11:02.576287 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e13a8cb-b751-4745-b780-f1db6ba5f2ff None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:02.593342 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-761003fa-6212-4768-85f8-1ecbeda60e17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:02.594749 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761003fa-6212-4768-85f8-1ecbeda60e17 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:02.595114 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761003fa-6212-4768-85f8-1ecbeda60e17 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:02.595459 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761003fa-6212-4768-85f8-1ecbeda60e17 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:02.596292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-761003fa-6212-4768-85f8-1ecbeda60e17 None admin] RBAC: Adding query filter params (role.id=bb9bd8ef24ff4aa5b639a2bd8567345b, scope.project.id=50a39f634ddf41798003dd395f14be68, user.id=4ed8fe9600cf453685d7c281db64fb87) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:02.596675 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-761003fa-6212-4768-85f8-1ecbeda60e17 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:02.598264 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-761003fa-6212-4768-85f8-1ecbeda60e17 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:02.608070 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 61/121] 10.222.0.42 () {56 vars in 1374 bytes} [Thu Oct 27 01:11:02 2022] GET /identity/v3/role_assignments?user.id=4ed8fe9600cf453685d7c281db64fb87&scope.project.id=50a39f634ddf41798003dd395f14be68&role.id=bb9bd8ef24ff4aa5b639a2bd8567345b => generated 608 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:02.608948 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-761003fa-6212-4768-85f8-1ecbeda60e17 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:02.739182 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e13a8cb-b751-4745-b780-f1db6ba5f2ff None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:02.750250 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-761003fa-6212-4768-85f8-1ecbeda60e17 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:02.785256 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:02.796161 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f740f0-105a-4039-9f1b-e2b043875a27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:02.796898 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f740f0-105a-4039-9f1b-e2b043875a27 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:02.797277 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f740f0-105a-4039-9f1b-e2b043875a27 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:02.800160 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f740f0-105a-4039-9f1b-e2b043875a27 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:02.800160 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f740f0-105a-4039-9f1b-e2b043875a27 None admin] RBAC: Adding query filter params (role.id=414286927fdd44688107ff5e10235acf, scope.project.id=50a39f634ddf41798003dd395f14be68, user.id=fe01809d18f644f9bc72669fc872b53d) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:02.800160 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f740f0-105a-4039-9f1b-e2b043875a27 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:02.800160 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f740f0-105a-4039-9f1b-e2b043875a27 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:02.804736 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ced1e53-dfa5-4ed1-aa2a-bf4a5975614f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:02.806121 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ced1e53-dfa5-4ed1-aa2a-bf4a5975614f None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:02.806704 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ced1e53-dfa5-4ed1-aa2a-bf4a5975614f None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:02.806985 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ced1e53-dfa5-4ed1-aa2a-bf4a5975614f None admin] PATH_INFO: `/v3/users/e72920d15dc445709a5d9c67d1cad0dc` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:02.814624 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 61/122] 10.222.0.42 () {56 vars in 1374 bytes} [Thu Oct 27 01:11:02 2022] GET /identity/v3/role_assignments?user.id=fe01809d18f644f9bc72669fc872b53d&scope.project.id=50a39f634ddf41798003dd395f14be68&role.id=414286927fdd44688107ff5e10235acf => generated 608 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:02.816777 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f740f0-105a-4039-9f1b-e2b043875a27 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:02.828461 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:02.832059 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ced1e53-dfa5-4ed1-aa2a-bf4a5975614f None admin] RBAC: Authorizing `identity:get_user(user_id=e72920d15dc445709a5d9c67d1cad0dc)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:02.834163 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ced1e53-dfa5-4ed1-aa2a-bf4a5975614f None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:02.836009 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 62/123] 10.222.0.42 () {56 vars in 1177 bytes} [Thu Oct 27 01:11:02 2022] GET /identity/v3/users/e72920d15dc445709a5d9c67d1cad0dc => generated 289 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:02.842649 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ced1e53-dfa5-4ed1-aa2a-bf4a5975614f None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:02.971734 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f740f0-105a-4039-9f1b-e2b043875a27 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:02.991916 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ced1e53-dfa5-4ed1-aa2a-bf4a5975614f None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:03.017796 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed56973-a41c-4b64-a6e0-e351a52b2a48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:03.018379 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed56973-a41c-4b64-a6e0-e351a52b2a48 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:03.018665 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed56973-a41c-4b64-a6e0-e351a52b2a48 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:03.019960 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed56973-a41c-4b64-a6e0-e351a52b2a48 None admin] PATH_INFO: `/v3/users/4ed8fe9600cf453685d7c281db64fb87` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:03.020328 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed56973-a41c-4b64-a6e0-e351a52b2a48 None admin] RBAC: Authorizing `identity:get_user(user_id=4ed8fe9600cf453685d7c281db64fb87)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:03.022272 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed56973-a41c-4b64-a6e0-e351a52b2a48 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:03.024635 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 62/124] 10.222.0.42 () {56 vars in 1177 bytes} [Thu Oct 27 01:11:02 2022] GET /identity/v3/users/4ed8fe9600cf453685d7c281db64fb87 => generated 275 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:03.033033 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed56973-a41c-4b64-a6e0-e351a52b2a48 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:03.047855 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce622cc7-ef35-49ee-a671-8c76c088e1c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:03.049575 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce622cc7-ef35-49ee-a671-8c76c088e1c8 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:03.050103 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce622cc7-ef35-49ee-a671-8c76c088e1c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:03.050578 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce622cc7-ef35-49ee-a671-8c76c088e1c8 None admin] PATH_INFO: `/v3/projects/50a39f634ddf41798003dd395f14be68` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:03.052723 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce622cc7-ef35-49ee-a671-8c76c088e1c8 None admin] RBAC: Authorizing `identity:get_project(project_id=50a39f634ddf41798003dd395f14be68)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:03.055764 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce622cc7-ef35-49ee-a671-8c76c088e1c8 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:03.058734 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 63/125] 10.222.0.42 () {56 vars in 1186 bytes} [Thu Oct 27 01:11:02 2022] GET /identity/v3/projects/50a39f634ddf41798003dd395f14be68 => generated 298 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:11:03.065241 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce622cc7-ef35-49ee-a671-8c76c088e1c8 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:03.134984 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:03.150576 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:03.197636 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed56973-a41c-4b64-a6e0-e351a52b2a48 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:03.199937 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce622cc7-ef35-49ee-a671-8c76c088e1c8 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:03.239386 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484bc883-1f4c-40d5-9f65-088cec5363e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:03.240061 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484bc883-1f4c-40d5-9f65-088cec5363e0 None admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:03.240344 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484bc883-1f4c-40d5-9f65-088cec5363e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:03.240614 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484bc883-1f4c-40d5-9f65-088cec5363e0 None admin] PATH_INFO: `/v3/projects/50a39f634ddf41798003dd395f14be68/users/e72920d15dc445709a5d9c67d1cad0dc/roles/5fa0c579f1fb428e9dcae929dd21846f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:03.242454 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03eaee8f-cfe7-415b-8377-bdfe67cbf6ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:03.242576 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-484bc883-1f4c-40d5-9f65-088cec5363e0 None admin] RBAC: Authorizing `identity:create_grant(project_id=50a39f634ddf41798003dd395f14be68, user_id=e72920d15dc445709a5d9c67d1cad0dc, role_id=5fa0c579f1fb428e9dcae929dd21846f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:03.243451 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03eaee8f-cfe7-415b-8377-bdfe67cbf6ec None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:03.243945 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03eaee8f-cfe7-415b-8377-bdfe67cbf6ec None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:03.244036 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-484bc883-1f4c-40d5-9f65-088cec5363e0 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:03.244224 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03eaee8f-cfe7-415b-8377-bdfe67cbf6ec None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:03.260697 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 64/126] 10.222.0.42 () {58 vars in 1439 bytes} [Thu Oct 27 01:11:03 2022] PUT /identity/v3/projects/50a39f634ddf41798003dd395f14be68/users/e72920d15dc445709a5d9c67d1cad0dc/roles/5fa0c579f1fb428e9dcae929dd21846f => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:11:03.327492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03eaee8f-cfe7-415b-8377-bdfe67cbf6ec None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:03.329013 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03eaee8f-cfe7-415b-8377-bdfe67cbf6ec None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:03.402379 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:03.451019 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 63/127] 10.222.0.42 () {58 vars in 1282 bytes} [Thu Oct 27 01:11:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 418 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 27 01:11:03.461334 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484bc883-1f4c-40d5-9f65-088cec5363e0 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:03.585269 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484bc883-1f4c-40d5-9f65-088cec5363e0 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:03.616966 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:03.628044 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d2d757-f37e-4352-9836-35817b6d199b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:03.628690 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d2d757-f37e-4352-9836-35817b6d199b None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:03.628973 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d2d757-f37e-4352-9836-35817b6d199b None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:03.629315 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d2d757-f37e-4352-9836-35817b6d199b None admin] PATH_INFO: `/v3/projects/d7d874be296c401193685c18ced01979` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:03.630524 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34d2d757-f37e-4352-9836-35817b6d199b None admin] RBAC: Authorizing `identity:get_project(project_id=d7d874be296c401193685c18ced01979)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:03.632029 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34d2d757-f37e-4352-9836-35817b6d199b None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:03.633911 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 65/128] 10.222.0.42 () {56 vars in 1186 bytes} [Thu Oct 27 01:11:03 2022] GET /identity/v3/projects/d7d874be296c401193685c18ced01979 => generated 312 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:03.639785 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03eaee8f-cfe7-415b-8377-bdfe67cbf6ec None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:03.748419 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03eaee8f-cfe7-415b-8377-bdfe67cbf6ec None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:03.783244 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd6ceca-3dd5-4d3f-9a47-1e2b9fb2179f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:03.783851 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd6ceca-3dd5-4d3f-9a47-1e2b9fb2179f None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:03.784135 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd6ceca-3dd5-4d3f-9a47-1e2b9fb2179f None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:03.784413 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd6ceca-3dd5-4d3f-9a47-1e2b9fb2179f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:03.785169 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd6ceca-3dd5-4d3f-9a47-1e2b9fb2179f None admin] RBAC: Adding query filter params (role.id=f199b79a7178462fa67bb8243943408f, scope.project.id=d7d874be296c401193685c18ced01979, user.id=4ed8fe9600cf453685d7c281db64fb87) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:03.785485 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd6ceca-3dd5-4d3f-9a47-1e2b9fb2179f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:03.787272 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd6ceca-3dd5-4d3f-9a47-1e2b9fb2179f None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:03.793403 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 64/129] 10.222.0.42 () {56 vars in 1374 bytes} [Thu Oct 27 01:11:03 2022] GET /identity/v3/role_assignments?user.id=4ed8fe9600cf453685d7c281db64fb87&scope.project.id=d7d874be296c401193685c18ced01979&role.id=f199b79a7178462fa67bb8243943408f => generated 608 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:04.970077 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f6e15b-02e4-4626-9928-3318abf28252 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:04.970578 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f6e15b-02e4-4626-9928-3318abf28252 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:04.970659 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f6e15b-02e4-4626-9928-3318abf28252 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:04.972123 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 66/130] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:04.977356 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4ca0bb-95a6-4cc0-90c2-a5f524a97fb4 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:04.977730 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4ca0bb-95a6-4cc0-90c2-a5f524a97fb4 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:04.978021 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4ca0bb-95a6-4cc0-90c2-a5f524a97fb4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:04.996788 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-6a4ca0bb-95a6-4cc0-90c2-a5f524a97fb4 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:04.998007 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-6a4ca0bb-95a6-4cc0-90c2-a5f524a97fb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:05.018294 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:05.065966 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 65/131] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:11:05.337947 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f6e15b-02e4-4626-9928-3318abf28252 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:05.444263 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f6e15b-02e4-4626-9928-3318abf28252 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:05.477532 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678b8cc8-fbae-4768-8f94-1b37b36f0136 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:05.478162 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678b8cc8-fbae-4768-8f94-1b37b36f0136 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:05.478441 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678b8cc8-fbae-4768-8f94-1b37b36f0136 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:05.478712 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678b8cc8-fbae-4768-8f94-1b37b36f0136 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:05.479811 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-678b8cc8-fbae-4768-8f94-1b37b36f0136 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:05.481275 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-678b8cc8-fbae-4768-8f94-1b37b36f0136 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:05.482838 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 67/132] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:11:05 2022] GET /identity/v3/domains/default => generated 199 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:05.488299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4ca0bb-95a6-4cc0-90c2-a5f524a97fb4 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:05.511666 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad8fe41-ad7e-4718-a41d-34927f092a38 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:05.511954 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad8fe41-ad7e-4718-a41d-34927f092a38 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:05.512230 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad8fe41-ad7e-4718-a41d-34927f092a38 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:05.513585 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 68/133] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 27 01:11:05.518368 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503e00dc-6b42-46d5-ae54-6edf7e3ac3b2 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:05.518652 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503e00dc-6b42-46d5-ae54-6edf7e3ac3b2 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:05.518926 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503e00dc-6b42-46d5-ae54-6edf7e3ac3b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:05.536816 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-503e00dc-6b42-46d5-ae54-6edf7e3ac3b2 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:05.538268 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-503e00dc-6b42-46d5-ae54-6edf7e3ac3b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:05.589646 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4ca0bb-95a6-4cc0-90c2-a5f524a97fb4 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:05.596986 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:05.607222 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 69/134] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 27 01:11:05.624362 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e23df5-5c05-4c09-926e-0b68aa75d212 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:05.624973 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e23df5-5c05-4c09-926e-0b68aa75d212 None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:05.625283 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e23df5-5c05-4c09-926e-0b68aa75d212 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:05.625559 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e23df5-5c05-4c09-926e-0b68aa75d212 None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:05.626220 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e23df5-5c05-4c09-926e-0b68aa75d212 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:05.627658 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e23df5-5c05-4c09-926e-0b68aa75d212 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:05.648374 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 66/135] 10.222.0.42 () {60 vars in 1132 bytes} [Thu Oct 27 01:11:05 2022] POST /identity/v3/users => generated 283 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:11:05.882074 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503e00dc-6b42-46d5-ae54-6edf7e3ac3b2 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:05.986365 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503e00dc-6b42-46d5-ae54-6edf7e3ac3b2 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:06.023831 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111e9f9e-1c68-4eaf-bf0d-577b2c023c6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:06.024463 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111e9f9e-1c68-4eaf-bf0d-577b2c023c6a None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:06.024741 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111e9f9e-1c68-4eaf-bf0d-577b2c023c6a None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:06.025012 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111e9f9e-1c68-4eaf-bf0d-577b2c023c6a None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:06.035605 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-111e9f9e-1c68-4eaf-bf0d-577b2c023c6a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:06.041110 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-111e9f9e-1c68-4eaf-bf0d-577b2c023c6a None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 27 01:11:06.041110 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:06.041110 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:06.041110 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:06.041110 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:06.041110 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:06.041110 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:06.041110 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:06.041110 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:06.041110 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:06.041110 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:06.041110 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:06.041110 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:06.041110 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:06.041110 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:06.041110 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:06.041975 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:06.041975 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:06.041975 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:06.041975 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:06.041975 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:06.041975 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:06.041975 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:06.041975 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:06.041975 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:06.041975 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:06.041975 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:06.041975 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:06.041975 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:06.041975 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:06.041975 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:06.041975 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:06.042750 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:06.042750 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:06.042750 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:06.042750 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:06.042750 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:06.042750 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:06.042750 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:06.042750 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:06.042750 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:06.042750 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:06.042750 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 27 01:11:06.042750 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:06.042750 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 70/136] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:11:05 2022] GET /identity/v3/roles/reader => generated 84 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:06.048631 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e23df5-5c05-4c09-926e-0b68aa75d212 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:06.065798 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:06.166127 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e23df5-5c05-4c09-926e-0b68aa75d212 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:06.202649 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9bfed0c-3de3-4f6c-bb02-0b352b880d05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:06.203260 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bfed0c-3de3-4f6c-bb02-0b352b880d05 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:06.203539 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bfed0c-3de3-4f6c-bb02-0b352b880d05 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:06.203808 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bfed0c-3de3-4f6c-bb02-0b352b880d05 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:06.204485 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9bfed0c-3de3-4f6c-bb02-0b352b880d05 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:06.204790 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9bfed0c-3de3-4f6c-bb02-0b352b880d05 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:06.206212 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9bfed0c-3de3-4f6c-bb02-0b352b880d05 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:06.219316 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 67/137] 10.222.0.42 () {56 vars in 1101 bytes} [Thu Oct 27 01:11:06 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:06.225760 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111e9f9e-1c68-4eaf-bf0d-577b2c023c6a None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:06.289920 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:06.337545 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111e9f9e-1c68-4eaf-bf0d-577b2c023c6a None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:06.373680 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a323b011-af8c-42ab-8e6e-04d08130f1de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:06.374693 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a323b011-af8c-42ab-8e6e-04d08130f1de None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:06.375166 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a323b011-af8c-42ab-8e6e-04d08130f1de None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:06.375648 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a323b011-af8c-42ab-8e6e-04d08130f1de None admin] PATH_INFO: `/v3/users/e72920d15dc445709a5d9c67d1cad0dc` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:06.376985 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a323b011-af8c-42ab-8e6e-04d08130f1de None admin] RBAC: Authorizing `identity:get_user(user_id=e72920d15dc445709a5d9c67d1cad0dc)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:06.378643 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a323b011-af8c-42ab-8e6e-04d08130f1de None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:06.380930 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 71/138] 10.222.0.42 () {56 vars in 1177 bytes} [Thu Oct 27 01:11:06 2022] GET /identity/v3/users/e72920d15dc445709a5d9c67d1cad0dc => generated 289 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:11:06.386314 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9bfed0c-3de3-4f6c-bb02-0b352b880d05 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:06.472444 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:06.501757 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9bfed0c-3de3-4f6c-bb02-0b352b880d05 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:06.538048 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a0cb62-3a66-411e-80fb-4f7722fb6ed2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:06.538668 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a0cb62-3a66-411e-80fb-4f7722fb6ed2 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:06.538952 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a0cb62-3a66-411e-80fb-4f7722fb6ed2 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:06.539225 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a0cb62-3a66-411e-80fb-4f7722fb6ed2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:06.610313 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a0cb62-3a66-411e-80fb-4f7722fb6ed2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:06.611843 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a0cb62-3a66-411e-80fb-4f7722fb6ed2 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:06.693360 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:06.721876 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 68/139] 10.222.0.42 () {58 vars in 1282 bytes} [Thu Oct 27 01:11:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 27 01:11:06.727625 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a323b011-af8c-42ab-8e6e-04d08130f1de None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:06.832860 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a323b011-af8c-42ab-8e6e-04d08130f1de None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:06.856764 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:06.870492 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d09a59-ec54-4339-9d56-ec3d0f8336da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:06.871129 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d09a59-ec54-4339-9d56-ec3d0f8336da None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:06.871406 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d09a59-ec54-4339-9d56-ec3d0f8336da None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:06.871680 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d09a59-ec54-4339-9d56-ec3d0f8336da None admin] PATH_INFO: `/v3/projects/50a39f634ddf41798003dd395f14be68` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:06.872813 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d09a59-ec54-4339-9d56-ec3d0f8336da None admin] RBAC: Authorizing `identity:get_project(project_id=50a39f634ddf41798003dd395f14be68)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:06.874299 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d09a59-ec54-4339-9d56-ec3d0f8336da None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:06.875926 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 72/140] 10.222.0.42 () {56 vars in 1186 bytes} [Thu Oct 27 01:11:06 2022] GET /identity/v3/projects/50a39f634ddf41798003dd395f14be68 => generated 298 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:06.882353 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a0cb62-3a66-411e-80fb-4f7722fb6ed2 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:06.987699 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a0cb62-3a66-411e-80fb-4f7722fb6ed2 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:07.022883 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ca75a4-5920-49e0-ac2d-0844b78a9aaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:07.023773 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ca75a4-5920-49e0-ac2d-0844b78a9aaf None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:07.024057 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ca75a4-5920-49e0-ac2d-0844b78a9aaf None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:07.024330 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ca75a4-5920-49e0-ac2d-0844b78a9aaf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:07.025076 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ca75a4-5920-49e0-ac2d-0844b78a9aaf None admin] RBAC: Adding query filter params (role.id=5fa0c579f1fb428e9dcae929dd21846f, scope.project.id=50a39f634ddf41798003dd395f14be68, user.id=e72920d15dc445709a5d9c67d1cad0dc) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:07.025400 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ca75a4-5920-49e0-ac2d-0844b78a9aaf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:07.026863 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ca75a4-5920-49e0-ac2d-0844b78a9aaf None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:07.032892 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 69/141] 10.222.0.42 () {56 vars in 1374 bytes} [Thu Oct 27 01:11:06 2022] GET /identity/v3/role_assignments?user.id=e72920d15dc445709a5d9c67d1cad0dc&scope.project.id=50a39f634ddf41798003dd395f14be68&role.id=5fa0c579f1fb428e9dcae929dd21846f => generated 608 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:07.976679 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3253aa1c-76d2-45ba-ac32-9e8ed483652c None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:07.977221 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3253aa1c-76d2-45ba-ac32-9e8ed483652c None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:07.977221 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3253aa1c-76d2-45ba-ac32-9e8ed483652c None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:07.978714 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 73/142] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:07.983768 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4b529e-c5b9-4ef9-95e5-1bef2437bc1c None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:07.984153 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4b529e-c5b9-4ef9-95e5-1bef2437bc1c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:07.984314 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4b529e-c5b9-4ef9-95e5-1bef2437bc1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:08.004532 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-3f4b529e-c5b9-4ef9-95e5-1bef2437bc1c None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:08.006657 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-3f4b529e-c5b9-4ef9-95e5-1bef2437bc1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:08.041900 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:08.082595 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 70/143] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:11:08.123092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bfcec41-a2fd-4df5-8d36-c7ef1798112c None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:08.123389 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bfcec41-a2fd-4df5-8d36-c7ef1798112c None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:08.123644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bfcec41-a2fd-4df5-8d36-c7ef1798112c None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:08.125161 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 74/144] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:08.130686 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f776bec2-3e68-416b-8f41-ee7b09d44a68 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:08.130975 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f776bec2-3e68-416b-8f41-ee7b09d44a68 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:08.131313 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f776bec2-3e68-416b-8f41-ee7b09d44a68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:08.149028 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-f776bec2-3e68-416b-8f41-ee7b09d44a68 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:08.150237 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f776bec2-3e68-416b-8f41-ee7b09d44a68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:08.198795 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b23c1d-0b42-4b31-abd2-62953c0d8302 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:08.199084 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b23c1d-0b42-4b31-abd2-62953c0d8302 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:08.199360 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b23c1d-0b42-4b31-abd2-62953c0d8302 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:08.200823 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 75/145] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 27 01:11:08.205518 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2163d1c9-bb2d-4391-9559-34b9fcde301a None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:08.205800 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2163d1c9-bb2d-4391-9559-34b9fcde301a None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:08.206074 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2163d1c9-bb2d-4391-9559-34b9fcde301a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:08.218144 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 71/146] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:11:08.227419 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-2163d1c9-bb2d-4391-9559-34b9fcde301a None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:08.228625 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-2163d1c9-bb2d-4391-9559-34b9fcde301a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:08.301757 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 76/147] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 27 01:11:08.362704 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f776bec2-3e68-416b-8f41-ee7b09d44a68 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:08.461201 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:08.465132 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f776bec2-3e68-416b-8f41-ee7b09d44a68 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:08.488182 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2163d1c9-bb2d-4391-9559-34b9fcde301a None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:08.502419 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0450ce51-f5ab-48e0-b418-153fb4576c21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:08.503042 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0450ce51-f5ab-48e0-b418-153fb4576c21 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:08.503522 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0450ce51-f5ab-48e0-b418-153fb4576c21 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:08.503795 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0450ce51-f5ab-48e0-b418-153fb4576c21 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:08.515071 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0450ce51-f5ab-48e0-b418-153fb4576c21 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:08.519910 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-0450ce51-f5ab-48e0-b418-153fb4576c21 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:11:08.519910 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:08.519910 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:08.519910 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:08.519910 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:08.519910 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:08.519910 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:08.519910 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:08.519910 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:08.519910 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:08.519910 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:08.519910 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:08.519910 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:08.519910 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:08.519910 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:08.519910 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:08.521618 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:08.521618 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:08.521618 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:08.521618 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:08.521618 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:08.521618 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:08.521618 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:08.521618 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:08.521618 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:08.521618 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:08.521618 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:08.521618 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:08.521618 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:08.521618 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:08.521618 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:08.521618 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:08.522583 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:08.522583 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:08.522583 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:08.522583 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:08.522583 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:08.522583 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:08.522583 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:08.522583 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:08.522583 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:08.522583 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:08.522583 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:11:08.522583 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:08.522583 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 72/148] 10.222.0.42 () {56 vars in 1096 bytes} [Thu Oct 27 01:11:08 2022] GET /identity/v3/roles/admin => generated 83 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:08.527930 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0450ce51-f5ab-48e0-b418-153fb4576c21 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:08.594209 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2163d1c9-bb2d-4391-9559-34b9fcde301a None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:08.613533 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:08.627576 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0450ce51-f5ab-48e0-b418-153fb4576c21 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:08.628342 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef619a6-ad46-4568-b5b1-ccc24cd628a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:08.628974 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef619a6-ad46-4568-b5b1-ccc24cd628a8 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:08.629275 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef619a6-ad46-4568-b5b1-ccc24cd628a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:08.629551 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef619a6-ad46-4568-b5b1-ccc24cd628a8 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:08.640136 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef619a6-ad46-4568-b5b1-ccc24cd628a8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:08.644926 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-6ef619a6-ad46-4568-b5b1-ccc24cd628a8 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 27 01:11:08.644926 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:08.644926 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:08.644926 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:08.644926 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:08.644926 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:08.644926 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:08.644926 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:08.644926 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:08.644926 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:08.644926 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:08.644926 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:08.644926 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:08.644926 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:08.644926 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:08.644926 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:08.645981 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:08.645981 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:08.645981 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:08.645981 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:08.645981 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:08.645981 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:08.645981 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:08.645981 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:08.645981 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:08.645981 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:08.645981 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:08.645981 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:08.645981 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:08.645981 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:08.645981 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:08.645981 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:08.646842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:08.646842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:08.646842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:08.646842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:08.646842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:08.646842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:08.646842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:08.646842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:08.646842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:08.646842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:08.646842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 27 01:11:08.646842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:08.646842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 77/149] 10.222.0.42 () {56 vars in 1114 bytes} [Thu Oct 27 01:11:08 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 27 01:11:08.647495 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef619a6-ad46-4568-b5b1-ccc24cd628a8 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:08.664665 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ca342d-630a-4f37-ad6e-db8514298233 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:08.665328 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ca342d-630a-4f37-ad6e-db8514298233 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:08.665610 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ca342d-630a-4f37-ad6e-db8514298233 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:08.665884 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ca342d-630a-4f37-ad6e-db8514298233 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:08.666560 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ca342d-630a-4f37-ad6e-db8514298233 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:08.666869 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ca342d-630a-4f37-ad6e-db8514298233 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:08.668247 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ca342d-630a-4f37-ad6e-db8514298233 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:08.681092 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 73/150] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:11:08 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:11:08.681092 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ca342d-630a-4f37-ad6e-db8514298233 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:08.701165 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:08.751290 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef619a6-ad46-4568-b5b1-ccc24cd628a8 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:08.777807 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ca342d-630a-4f37-ad6e-db8514298233 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:08.784314 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598737c0-f0ee-4e90-b847-d58cf39c48d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:08.784939 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598737c0-f0ee-4e90-b847-d58cf39c48d1 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:08.785254 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598737c0-f0ee-4e90-b847-d58cf39c48d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:08.785525 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598737c0-f0ee-4e90-b847-d58cf39c48d1 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:08.786785 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598737c0-f0ee-4e90-b847-d58cf39c48d1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:08.788239 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598737c0-f0ee-4e90-b847-d58cf39c48d1 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:08.789829 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 78/151] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:11:08 2022] GET /identity/v3/domains/default => generated 199 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:08.790665 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598737c0-f0ee-4e90-b847-d58cf39c48d1 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:08.820437 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38fe06ec-b150-4b72-ba6e-5757ba2b0354 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:08.821095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38fe06ec-b150-4b72-ba6e-5757ba2b0354 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:08.821376 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38fe06ec-b150-4b72-ba6e-5757ba2b0354 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:08.821646 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38fe06ec-b150-4b72-ba6e-5757ba2b0354 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:08.822313 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38fe06ec-b150-4b72-ba6e-5757ba2b0354 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:08.822614 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38fe06ec-b150-4b72-ba6e-5757ba2b0354 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:08.823992 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38fe06ec-b150-4b72-ba6e-5757ba2b0354 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:08.831773 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 74/152] 10.222.0.42 () {56 vars in 1111 bytes} [Thu Oct 27 01:11:08 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:08.832612 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38fe06ec-b150-4b72-ba6e-5757ba2b0354 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:08.843776 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:08.895060 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598737c0-f0ee-4e90-b847-d58cf39c48d1 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:08.913595 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:08.928511 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c29565-8ff6-402b-9ed0-37511625ae9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:08.929168 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c29565-8ff6-402b-9ed0-37511625ae9d None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:08.929446 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c29565-8ff6-402b-9ed0-37511625ae9d None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:08.929714 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c29565-8ff6-402b-9ed0-37511625ae9d None admin] PATH_INFO: `/v3/users/04afced9034d4384ada854ebde4960f5` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:08.933600 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38fe06ec-b150-4b72-ba6e-5757ba2b0354 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:08.943391 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c29565-8ff6-402b-9ed0-37511625ae9d None admin] RBAC: Authorizing `identity:get_user(user_id=04afced9034d4384ada854ebde4960f5)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:08.944837 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c29565-8ff6-402b-9ed0-37511625ae9d None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:08.946661 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 79/153] 10.222.0.42 () {56 vars in 1177 bytes} [Thu Oct 27 01:11:08 2022] GET /identity/v3/users/04afced9034d4384ada854ebde4960f5 => generated 283 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:08.947220 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c29565-8ff6-402b-9ed0-37511625ae9d None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:08.970121 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02f3d65-27da-454b-832c-01843cefb336 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:08.970714 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02f3d65-27da-454b-832c-01843cefb336 None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:08.970991 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02f3d65-27da-454b-832c-01843cefb336 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:08.971263 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02f3d65-27da-454b-832c-01843cefb336 None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:08.971909 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02f3d65-27da-454b-832c-01843cefb336 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:08.973377 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02f3d65-27da-454b-832c-01843cefb336 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:08.994510 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 75/154] 10.222.0.42 () {60 vars in 1132 bytes} [Thu Oct 27 01:11:08 2022] POST /identity/v3/users => generated 297 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:11:09.007193 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02f3d65-27da-454b-832c-01843cefb336 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:09.054738 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c29565-8ff6-402b-9ed0-37511625ae9d None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:09.063223 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:09.096930 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4552b20e-972e-477c-8fb2-2a00c62c5c0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:09.097769 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4552b20e-972e-477c-8fb2-2a00c62c5c0d None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:09.097935 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4552b20e-972e-477c-8fb2-2a00c62c5c0d None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:09.098194 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4552b20e-972e-477c-8fb2-2a00c62c5c0d None admin] PATH_INFO: `/v3/users/04afced9034d4384ada854ebde4960f5` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:09.100035 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4552b20e-972e-477c-8fb2-2a00c62c5c0d None admin] RBAC: Authorizing `identity:get_user(user_id=04afced9034d4384ada854ebde4960f5)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:09.101681 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4552b20e-972e-477c-8fb2-2a00c62c5c0d None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:09.103424 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 80/155] 10.222.0.42 () {56 vars in 1177 bytes} [Thu Oct 27 01:11:08 2022] GET /identity/v3/users/04afced9034d4384ada854ebde4960f5 => generated 283 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:09.109043 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4552b20e-972e-477c-8fb2-2a00c62c5c0d None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:09.127374 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02f3d65-27da-454b-832c-01843cefb336 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:09.167147 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b6780f-8996-4813-b41a-0f87ff2a502c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:09.167793 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b6780f-8996-4813-b41a-0f87ff2a502c None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:09.168066 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b6780f-8996-4813-b41a-0f87ff2a502c None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:09.168478 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b6780f-8996-4813-b41a-0f87ff2a502c None admin] PATH_INFO: `/v3/projects/69bfdc060ff949e2ba795cad23f9f746` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:09.169652 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b6780f-8996-4813-b41a-0f87ff2a502c None admin] RBAC: Authorizing `identity:get_project(project_id=69bfdc060ff949e2ba795cad23f9f746)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:09.171098 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b6780f-8996-4813-b41a-0f87ff2a502c None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:09.172901 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 76/156] 10.222.0.42 () {56 vars in 1186 bytes} [Thu Oct 27 01:11:09 2022] GET /identity/v3/projects/69bfdc060ff949e2ba795cad23f9f746 => generated 302 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:09.179549 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b6780f-8996-4813-b41a-0f87ff2a502c None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:09.191057 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:09.230043 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4552b20e-972e-477c-8fb2-2a00c62c5c0d None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:09.272075 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946ab4cc-3908-45df-9b82-8ca3f78744a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:09.273025 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946ab4cc-3908-45df-9b82-8ca3f78744a9 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:09.273383 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946ab4cc-3908-45df-9b82-8ca3f78744a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:09.273716 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946ab4cc-3908-45df-9b82-8ca3f78744a9 None admin] PATH_INFO: `/v3/projects/69bfdc060ff949e2ba795cad23f9f746` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:09.276940 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946ab4cc-3908-45df-9b82-8ca3f78744a9 None admin] RBAC: Authorizing `identity:get_project(project_id=69bfdc060ff949e2ba795cad23f9f746)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:09.277360 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946ab4cc-3908-45df-9b82-8ca3f78744a9 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:09.279027 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 81/157] 10.222.0.42 () {56 vars in 1186 bytes} [Thu Oct 27 01:11:09 2022] GET /identity/v3/projects/69bfdc060ff949e2ba795cad23f9f746 => generated 302 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:11:09.285619 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946ab4cc-3908-45df-9b82-8ca3f78744a9 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:09.309120 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b6780f-8996-4813-b41a-0f87ff2a502c None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:09.341280 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:09.349309 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6855539f-f612-41fc-804c-a8189d3041f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:09.350128 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6855539f-f612-41fc-804c-a8189d3041f3 None admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:09.351119 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6855539f-f612-41fc-804c-a8189d3041f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:09.351119 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6855539f-f612-41fc-804c-a8189d3041f3 None admin] PATH_INFO: `/v3/projects/69bfdc060ff949e2ba795cad23f9f746/users/04afced9034d4384ada854ebde4960f5/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:09.353095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6855539f-f612-41fc-804c-a8189d3041f3 None admin] RBAC: Authorizing `identity:create_grant(project_id=69bfdc060ff949e2ba795cad23f9f746, user_id=04afced9034d4384ada854ebde4960f5, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:09.354657 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6855539f-f612-41fc-804c-a8189d3041f3 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:09.373312 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 77/158] 10.222.0.42 () {58 vars in 1439 bytes} [Thu Oct 27 01:11:09 2022] PUT /identity/v3/projects/69bfdc060ff949e2ba795cad23f9f746/users/04afced9034d4384ada854ebde4960f5/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:11:09.418416 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946ab4cc-3908-45df-9b82-8ca3f78744a9 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:09.461008 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54d7054-8d8a-410c-bd88-2af550003367 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:09.461885 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54d7054-8d8a-410c-bd88-2af550003367 None admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:09.462276 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54d7054-8d8a-410c-bd88-2af550003367 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:09.462680 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54d7054-8d8a-410c-bd88-2af550003367 None admin] PATH_INFO: `/v3/projects/69bfdc060ff949e2ba795cad23f9f746/users/04afced9034d4384ada854ebde4960f5/roles/bb9bd8ef24ff4aa5b639a2bd8567345b` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:09.465196 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54d7054-8d8a-410c-bd88-2af550003367 None admin] RBAC: Authorizing `identity:create_grant(project_id=69bfdc060ff949e2ba795cad23f9f746, user_id=04afced9034d4384ada854ebde4960f5, role_id=bb9bd8ef24ff4aa5b639a2bd8567345b)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:09.466804 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54d7054-8d8a-410c-bd88-2af550003367 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:09.488071 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 82/159] 10.222.0.42 () {58 vars in 1439 bytes} [Thu Oct 27 01:11:09 2022] PUT /identity/v3/projects/69bfdc060ff949e2ba795cad23f9f746/users/04afced9034d4384ada854ebde4960f5/roles/bb9bd8ef24ff4aa5b639a2bd8567345b => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:11:11.336868 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49cdae4e-c5cf-46b9-8f99-0528d3c42d0a None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:11.337439 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49cdae4e-c5cf-46b9-8f99-0528d3c42d0a None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:11.337439 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49cdae4e-c5cf-46b9-8f99-0528d3c42d0a None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:11.338813 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 78/160] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:11.344012 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2e7bda-94b2-4693-8351-3b6485db01c1 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:11.344532 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2e7bda-94b2-4693-8351-3b6485db01c1 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:11.344817 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2e7bda-94b2-4693-8351-3b6485db01c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:11.366961 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-6d2e7bda-94b2-4693-8351-3b6485db01c1 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:11.368147 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-6d2e7bda-94b2-4693-8351-3b6485db01c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:11.423802 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f6b3cf-5c77-4f2e-b03a-976cd1cbdc61 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:11.424100 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f6b3cf-5c77-4f2e-b03a-976cd1cbdc61 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:11.424341 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f6b3cf-5c77-4f2e-b03a-976cd1cbdc61 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:11.425857 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 79/161] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:11.431135 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f609dcad-92ca-400a-94c4-0efe37bba450 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:11.431672 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f609dcad-92ca-400a-94c4-0efe37bba450 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:11.431994 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f609dcad-92ca-400a-94c4-0efe37bba450 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:11.441789 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 83/162] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:11:11.457213 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-f609dcad-92ca-400a-94c4-0efe37bba450 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:11.458995 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f609dcad-92ca-400a-94c4-0efe37bba450 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:11.523007 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:11.530889 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 80/163] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:11:11.569032 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18871a4-cc01-4b30-b727-bd6d8cad0ba7 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:11.569350 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18871a4-cc01-4b30-b727-bd6d8cad0ba7 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:11.569622 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18871a4-cc01-4b30-b727-bd6d8cad0ba7 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:11.571252 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 84/164] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:11.576508 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4019e3c-e727-459f-bcf6-ee6aae3f6e26 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:11.576802 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4019e3c-e727-459f-bcf6-ee6aae3f6e26 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:11.577090 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4019e3c-e727-459f-bcf6-ee6aae3f6e26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:11.595591 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-f4019e3c-e727-459f-bcf6-ee6aae3f6e26 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:11.596704 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f4019e3c-e727-459f-bcf6-ee6aae3f6e26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:11.674640 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 81/165] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:11:11.725689 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18871a4-cc01-4b30-b727-bd6d8cad0ba7 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:11.733724 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40fb2241-cf23-47aa-8bd7-c1af8506b2b4 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:11.734055 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40fb2241-cf23-47aa-8bd7-c1af8506b2b4 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:11.734287 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40fb2241-cf23-47aa-8bd7-c1af8506b2b4 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:11.735794 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 82/166] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 27 01:11:11.742873 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f47dd85-3d64-4d91-b26e-0cbc7620499d None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:11.743158 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f47dd85-3d64-4d91-b26e-0cbc7620499d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:11.743439 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f47dd85-3d64-4d91-b26e-0cbc7620499d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:11.761076 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:11.765164 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-9f47dd85-3d64-4d91-b26e-0cbc7620499d None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:11.766920 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-9f47dd85-3d64-4d91-b26e-0cbc7620499d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:11.851104 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18871a4-cc01-4b30-b727-bd6d8cad0ba7 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:11.853138 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 83/167] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 27 01:11:11.853138 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f47dd85-3d64-4d91-b26e-0cbc7620499d None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:11.894161 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce93a245-73a6-4b4a-ab65-4cf67279ba36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:11.894783 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce93a245-73a6-4b4a-ab65-4cf67279ba36 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:11.895057 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce93a245-73a6-4b4a-ab65-4cf67279ba36 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:11.895327 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce93a245-73a6-4b4a-ab65-4cf67279ba36 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:11.908708 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce93a245-73a6-4b4a-ab65-4cf67279ba36 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:11.915343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-ce93a245-73a6-4b4a-ab65-4cf67279ba36 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 27 01:11:11.915343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:11.915343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:11.915343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:11.915343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:11.915343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:11.915343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:11.915343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:11.915343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:11.915343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:11.915343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:11.915343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:11.915343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:11.915343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:11.915343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:11.915343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:11.916448 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:11.916448 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:11.916448 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:11.916448 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:11.916448 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:11.916448 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:11.916448 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:11.916448 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:11.916448 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:11.916448 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:11.916448 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:11.916448 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:11.916448 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:11.916448 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:11.916448 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:11.916448 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:11.917479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:11.917479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:11.917479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:11.917479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:11.917479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:11.917479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:11.917479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:11.917479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:11.917479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:11.917479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:11.917479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 27 01:11:11.917479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:11.917479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 85/168] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:11:11 2022] GET /identity/v3/roles/member => generated 84 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:11.921597 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:11.922433 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce93a245-73a6-4b4a-ab65-4cf67279ba36 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:11.978195 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f47dd85-3d64-4d91-b26e-0cbc7620499d None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:12.014175 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e500c58-84bc-47e3-b63c-671ab9e63e8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:12.014831 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e500c58-84bc-47e3-b63c-671ab9e63e8c None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:12.015106 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e500c58-84bc-47e3-b63c-671ab9e63e8c None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:12.015374 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e500c58-84bc-47e3-b63c-671ab9e63e8c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:12.016744 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:12.017425 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e500c58-84bc-47e3-b63c-671ab9e63e8c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:12.018883 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e500c58-84bc-47e3-b63c-671ab9e63e8c None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:12.020594 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 84/169] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:11:11 2022] GET /identity/v3/domains/default => generated 199 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:12.021499 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e500c58-84bc-47e3-b63c-671ab9e63e8c None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:12.030955 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce93a245-73a6-4b4a-ab65-4cf67279ba36 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:12.066320 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f06580-8ca6-45a1-a8e1-1e25e314381c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:12.066921 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f06580-8ca6-45a1-a8e1-1e25e314381c None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:12.067212 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f06580-8ca6-45a1-a8e1-1e25e314381c None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:12.068088 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f06580-8ca6-45a1-a8e1-1e25e314381c None admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:12.068900 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f06580-8ca6-45a1-a8e1-1e25e314381c None admin] RBAC: Adding query filter params (name=member) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:12.069228 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f06580-8ca6-45a1-a8e1-1e25e314381c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:12.070609 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f06580-8ca6-45a1-a8e1-1e25e314381c None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:12.079218 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 86/170] 10.222.0.42 () {56 vars in 1101 bytes} [Thu Oct 27 01:11:11 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:11:12.080011 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f06580-8ca6-45a1-a8e1-1e25e314381c None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:12.137925 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e500c58-84bc-47e3-b63c-671ab9e63e8c None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:12.160762 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:12.172423 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2059fe9-4c54-40eb-bb34-a3e3717039a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:12.173051 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2059fe9-4c54-40eb-bb34-a3e3717039a2 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:12.173350 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2059fe9-4c54-40eb-bb34-a3e3717039a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:12.173619 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2059fe9-4c54-40eb-bb34-a3e3717039a2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:12.182524 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f06580-8ca6-45a1-a8e1-1e25e314381c None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:12.184348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2059fe9-4c54-40eb-bb34-a3e3717039a2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:12.189206 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-a2059fe9-4c54-40eb-bb34-a3e3717039a2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:11:12.189206 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:12.189206 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:12.189206 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:12.189206 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:12.189206 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:12.189206 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:12.189206 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:12.189206 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:12.189206 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:12.189206 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:12.189206 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:12.189206 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:12.189206 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:12.189206 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:12.189206 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:12.190078 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:12.190078 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:12.190078 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:12.190078 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:12.190078 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:12.190078 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:12.190078 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:12.190078 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:12.190078 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:12.190078 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:12.190078 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:12.190078 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:12.190078 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:12.190078 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:12.190078 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:12.190078 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:12.190853 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:12.190853 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:12.190853 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:12.190853 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:12.190853 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:12.190853 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:12.190853 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:12.190853 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:12.190853 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:12.190853 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:12.190853 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:11:12.190853 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:12.190853 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 85/171] 10.222.0.42 () {56 vars in 1096 bytes} [Thu Oct 27 01:11:12 2022] GET /identity/v3/roles/admin => generated 83 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:12.191500 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2059fe9-4c54-40eb-bb34-a3e3717039a2 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:12.217158 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7029ff-de51-43f5-ac0f-bd67ca3950ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:12.217773 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7029ff-de51-43f5-ac0f-bd67ca3950ff None admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:12.218048 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7029ff-de51-43f5-ac0f-bd67ca3950ff None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:12.218318 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7029ff-de51-43f5-ac0f-bd67ca3950ff None admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:12.218983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee7029ff-de51-43f5-ac0f-bd67ca3950ff None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:12.220420 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee7029ff-de51-43f5-ac0f-bd67ca3950ff None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:12.242304 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 87/172] 10.222.0.42 () {60 vars in 1132 bytes} [Thu Oct 27 01:11:12 2022] POST /identity/v3/users => generated 297 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:11:12.243106 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7029ff-de51-43f5-ac0f-bd67ca3950ff None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:12.258264 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:12.307295 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2059fe9-4c54-40eb-bb34-a3e3717039a2 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:12.347161 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371a1347-f2ef-4022-81a8-5ef4c70562ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:12.347831 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371a1347-f2ef-4022-81a8-5ef4c70562ad None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:12.348120 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371a1347-f2ef-4022-81a8-5ef4c70562ad None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:12.348372 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371a1347-f2ef-4022-81a8-5ef4c70562ad None admin] PATH_INFO: `/v3/users/c40599d01dd34fb8a1bf426d02fd12e2` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:12.364451 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7029ff-de51-43f5-ac0f-bd67ca3950ff None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:12.370462 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371a1347-f2ef-4022-81a8-5ef4c70562ad None admin] RBAC: Authorizing `identity:get_user(user_id=c40599d01dd34fb8a1bf426d02fd12e2)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:12.372165 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371a1347-f2ef-4022-81a8-5ef4c70562ad None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:12.378686 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 86/173] 10.222.0.42 () {56 vars in 1177 bytes} [Thu Oct 27 01:11:12 2022] GET /identity/v3/users/c40599d01dd34fb8a1bf426d02fd12e2 => generated 297 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:12.378686 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371a1347-f2ef-4022-81a8-5ef4c70562ad None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:12.408420 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9465e1b6-7d08-4c4c-bd95-f5bcde4a156e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:12.409099 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9465e1b6-7d08-4c4c-bd95-f5bcde4a156e None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:12.409378 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9465e1b6-7d08-4c4c-bd95-f5bcde4a156e None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:12.409645 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9465e1b6-7d08-4c4c-bd95-f5bcde4a156e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:12.421660 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9465e1b6-7d08-4c4c-bd95-f5bcde4a156e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:12.427608 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-9465e1b6-7d08-4c4c-bd95-f5bcde4a156e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 27 01:11:12.427608 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:12.427608 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:12.427608 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:12.427608 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:12.427608 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:12.427608 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:12.427608 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:12.427608 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:12.427608 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:12.427608 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:12.427608 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:12.427608 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:12.427608 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:12.427608 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:12.427608 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:12.428583 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:12.428583 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:12.428583 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:12.428583 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:12.428583 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:12.428583 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:12.428583 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:12.428583 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:12.428583 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:12.428583 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:12.428583 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:12.428583 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:12.428583 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:12.428583 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:12.428583 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:12.428583 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:12.434086 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:12.434086 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:12.434086 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:12.434086 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:12.434086 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:12.434086 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:12.434086 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:12.434086 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:12.434086 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:12.434086 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:12.434086 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 27 01:11:12.434086 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:12.434086 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 88/174] 10.222.0.42 () {56 vars in 1114 bytes} [Thu Oct 27 01:11:12 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:12.434086 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9465e1b6-7d08-4c4c-bd95-f5bcde4a156e None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:12.436147 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:12.504859 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371a1347-f2ef-4022-81a8-5ef4c70562ad None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:12.508932 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:12.554759 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83ecad2-9eaa-4695-aff6-e432656e1b47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:12.555441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83ecad2-9eaa-4695-aff6-e432656e1b47 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:12.555605 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9465e1b6-7d08-4c4c-bd95-f5bcde4a156e None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:12.555683 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83ecad2-9eaa-4695-aff6-e432656e1b47 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:12.555907 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83ecad2-9eaa-4695-aff6-e432656e1b47 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:12.556835 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83ecad2-9eaa-4695-aff6-e432656e1b47 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:12.557183 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83ecad2-9eaa-4695-aff6-e432656e1b47 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:12.558553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83ecad2-9eaa-4695-aff6-e432656e1b47 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:12.569788 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 87/175] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:11:12 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:12.570587 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83ecad2-9eaa-4695-aff6-e432656e1b47 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:12.604151 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf3f67b-8640-4120-b02e-62eef1ee786a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:12.604786 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf3f67b-8640-4120-b02e-62eef1ee786a None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:12.605253 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf3f67b-8640-4120-b02e-62eef1ee786a None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:12.605448 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf3f67b-8640-4120-b02e-62eef1ee786a None admin] PATH_INFO: `/v3/projects/69bfdc060ff949e2ba795cad23f9f746` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:12.606821 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf3f67b-8640-4120-b02e-62eef1ee786a None admin] RBAC: Authorizing `identity:get_project(project_id=69bfdc060ff949e2ba795cad23f9f746)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:12.608355 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf3f67b-8640-4120-b02e-62eef1ee786a None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:12.610326 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 89/176] 10.222.0.42 () {56 vars in 1186 bytes} [Thu Oct 27 01:11:12 2022] GET /identity/v3/projects/69bfdc060ff949e2ba795cad23f9f746 => generated 302 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:12.610852 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf3f67b-8640-4120-b02e-62eef1ee786a None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:12.699307 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83ecad2-9eaa-4695-aff6-e432656e1b47 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:12.701868 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:12.725662 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf3f67b-8640-4120-b02e-62eef1ee786a None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:12.740175 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-321a98cd-9ea8-4936-8471-d48e2f8fa93b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:12.740832 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321a98cd-9ea8-4936-8471-d48e2f8fa93b None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:12.741138 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321a98cd-9ea8-4936-8471-d48e2f8fa93b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:12.741415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321a98cd-9ea8-4936-8471-d48e2f8fa93b None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:12.742094 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-321a98cd-9ea8-4936-8471-d48e2f8fa93b None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:12.742523 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-321a98cd-9ea8-4936-8471-d48e2f8fa93b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:12.744010 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-321a98cd-9ea8-4936-8471-d48e2f8fa93b None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:12.753393 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 88/177] 10.222.0.42 () {56 vars in 1111 bytes} [Thu Oct 27 01:11:12 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:12.754120 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-321a98cd-9ea8-4936-8471-d48e2f8fa93b None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:12.761187 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cee0f9a-e5e9-4bb3-9f61-8d91fd3d8386 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:12.761828 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cee0f9a-e5e9-4bb3-9f61-8d91fd3d8386 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:12.762107 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cee0f9a-e5e9-4bb3-9f61-8d91fd3d8386 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:12.762381 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cee0f9a-e5e9-4bb3-9f61-8d91fd3d8386 None admin] PATH_INFO: `/v3/users/04afced9034d4384ada854ebde4960f5` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:12.763529 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cee0f9a-e5e9-4bb3-9f61-8d91fd3d8386 None admin] RBAC: Authorizing `identity:get_user(user_id=04afced9034d4384ada854ebde4960f5)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:12.764978 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cee0f9a-e5e9-4bb3-9f61-8d91fd3d8386 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:12.766856 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 90/178] 10.222.0.42 () {56 vars in 1177 bytes} [Thu Oct 27 01:11:12 2022] GET /identity/v3/users/04afced9034d4384ada854ebde4960f5 => generated 283 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:12.767682 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cee0f9a-e5e9-4bb3-9f61-8d91fd3d8386 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:12.776863 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:12.871396 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-321a98cd-9ea8-4936-8471-d48e2f8fa93b None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:12.871587 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cee0f9a-e5e9-4bb3-9f61-8d91fd3d8386 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:12.907147 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225eddc2-9649-4da9-8d18-88c22d60db8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:12.907786 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225eddc2-9649-4da9-8d18-88c22d60db8f None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:12.908062 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225eddc2-9649-4da9-8d18-88c22d60db8f None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:12.908335 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225eddc2-9649-4da9-8d18-88c22d60db8f None admin] PATH_INFO: `/v3/users/04afced9034d4384ada854ebde4960f5` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:12.909491 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-225eddc2-9649-4da9-8d18-88c22d60db8f None admin] RBAC: Authorizing `identity:get_user(user_id=04afced9034d4384ada854ebde4960f5)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:12.910610 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0779eb6-ff11-4be0-8687-318093129dcf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:12.911501 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-225eddc2-9649-4da9-8d18-88c22d60db8f None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:12.911798 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0779eb6-ff11-4be0-8687-318093129dcf None admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:12.912077 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0779eb6-ff11-4be0-8687-318093129dcf None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:12.912346 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0779eb6-ff11-4be0-8687-318093129dcf None admin] PATH_INFO: `/v3/projects/69bfdc060ff949e2ba795cad23f9f746/users/c40599d01dd34fb8a1bf426d02fd12e2/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:12.913038 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 91/179] 10.222.0.42 () {56 vars in 1177 bytes} [Thu Oct 27 01:11:12 2022] GET /identity/v3/users/04afced9034d4384ada854ebde4960f5 => generated 283 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:12.913815 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225eddc2-9649-4da9-8d18-88c22d60db8f None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:12.914547 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0779eb6-ff11-4be0-8687-318093129dcf None admin] RBAC: Authorizing `identity:create_grant(project_id=69bfdc060ff949e2ba795cad23f9f746, user_id=c40599d01dd34fb8a1bf426d02fd12e2, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:12.916004 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0779eb6-ff11-4be0-8687-318093129dcf None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:12.932791 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 89/180] 10.222.0.42 () {58 vars in 1439 bytes} [Thu Oct 27 01:11:12 2022] PUT /identity/v3/projects/69bfdc060ff949e2ba795cad23f9f746/users/c40599d01dd34fb8a1bf426d02fd12e2/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:11:12.933897 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0779eb6-ff11-4be0-8687-318093129dcf None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:12.968136 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:12.996610 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:13.057337 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225eddc2-9649-4da9-8d18-88c22d60db8f None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:13.063984 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0779eb6-ff11-4be0-8687-318093129dcf None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:13.101516 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb7a594-04aa-4d87-95ea-ac330c61ebb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:13.102121 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb7a594-04aa-4d87-95ea-ac330c61ebb3 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:13.102397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb7a594-04aa-4d87-95ea-ac330c61ebb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:13.102685 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb7a594-04aa-4d87-95ea-ac330c61ebb3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:13.105042 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a08c4ed-4516-49d4-931f-498cd5e0af3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:13.105982 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a08c4ed-4516-49d4-931f-498cd5e0af3b None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:13.106605 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a08c4ed-4516-49d4-931f-498cd5e0af3b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:13.107057 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a08c4ed-4516-49d4-931f-498cd5e0af3b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:13.185636 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a08c4ed-4516-49d4-931f-498cd5e0af3b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:13.187664 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a08c4ed-4516-49d4-931f-498cd5e0af3b None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:13.190332 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb7a594-04aa-4d87-95ea-ac330c61ebb3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:13.191891 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb7a594-04aa-4d87-95ea-ac330c61ebb3 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:13.217701 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:13.272895 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:13.301270 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 92/181] 10.222.0.42 () {58 vars in 1282 bytes} [Thu Oct 27 01:11:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 388 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 27 01:11:13.304456 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 90/182] 10.222.0.42 () {58 vars in 1282 bytes} [Thu Oct 27 01:11:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 27 01:11:13.307373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb7a594-04aa-4d87-95ea-ac330c61ebb3 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:13.310725 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a08c4ed-4516-49d4-931f-498cd5e0af3b None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:13.424496 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a08c4ed-4516-49d4-931f-498cd5e0af3b None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:13.456450 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:13.467442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be87fa0a-0cbc-4f2e-9bcb-e93a896c879d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:13.468368 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be87fa0a-0cbc-4f2e-9bcb-e93a896c879d None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:13.468859 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be87fa0a-0cbc-4f2e-9bcb-e93a896c879d None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:13.469340 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be87fa0a-0cbc-4f2e-9bcb-e93a896c879d None admin] PATH_INFO: `/v3/projects/69bfdc060ff949e2ba795cad23f9f746` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:13.470658 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be87fa0a-0cbc-4f2e-9bcb-e93a896c879d None admin] RBAC: Authorizing `identity:get_project(project_id=69bfdc060ff949e2ba795cad23f9f746)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:13.472280 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be87fa0a-0cbc-4f2e-9bcb-e93a896c879d None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:13.474629 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 91/183] 10.222.0.42 () {56 vars in 1186 bytes} [Thu Oct 27 01:11:13 2022] GET /identity/v3/projects/69bfdc060ff949e2ba795cad23f9f746 => generated 302 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:11:13.483271 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be87fa0a-0cbc-4f2e-9bcb-e93a896c879d None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:13.485623 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb7a594-04aa-4d87-95ea-ac330c61ebb3 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:13.540102 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a86dbf-448e-476f-a23e-ef9cd6c679a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:13.540671 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a86dbf-448e-476f-a23e-ef9cd6c679a5 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:13.540950 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a86dbf-448e-476f-a23e-ef9cd6c679a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:13.541260 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a86dbf-448e-476f-a23e-ef9cd6c679a5 None admin] PATH_INFO: `/v3/projects/69bfdc060ff949e2ba795cad23f9f746` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:13.546232 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a86dbf-448e-476f-a23e-ef9cd6c679a5 None admin] RBAC: Authorizing `identity:get_project(project_id=69bfdc060ff949e2ba795cad23f9f746)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:13.547810 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a86dbf-448e-476f-a23e-ef9cd6c679a5 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:13.549491 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 93/184] 10.222.0.42 () {56 vars in 1186 bytes} [Thu Oct 27 01:11:13 2022] GET /identity/v3/projects/69bfdc060ff949e2ba795cad23f9f746 => generated 302 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:11:13.555674 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a86dbf-448e-476f-a23e-ef9cd6c679a5 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:13.625564 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:13.668054 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be87fa0a-0cbc-4f2e-9bcb-e93a896c879d None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:13.730322 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b297b85c-ff92-4b3e-a37c-50fdefb3ef18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:13.730941 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b297b85c-ff92-4b3e-a37c-50fdefb3ef18 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:13.731220 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b297b85c-ff92-4b3e-a37c-50fdefb3ef18 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:13.731494 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b297b85c-ff92-4b3e-a37c-50fdefb3ef18 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:13.732288 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b297b85c-ff92-4b3e-a37c-50fdefb3ef18 None admin] RBAC: Adding query filter params (role.id=bb9bd8ef24ff4aa5b639a2bd8567345b, scope.project.id=69bfdc060ff949e2ba795cad23f9f746, user.id=04afced9034d4384ada854ebde4960f5) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:13.732603 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b297b85c-ff92-4b3e-a37c-50fdefb3ef18 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:13.734088 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b297b85c-ff92-4b3e-a37c-50fdefb3ef18 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:13.737369 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a86dbf-448e-476f-a23e-ef9cd6c679a5 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:13.750170 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 92/185] 10.222.0.42 () {56 vars in 1374 bytes} [Thu Oct 27 01:11:13 2022] GET /identity/v3/role_assignments?user.id=04afced9034d4384ada854ebde4960f5&scope.project.id=69bfdc060ff949e2ba795cad23f9f746&role.id=bb9bd8ef24ff4aa5b639a2bd8567345b => generated 608 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:13.784110 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c58a9294-30c7-4e10-b71d-c88ec7dae0f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:13.784727 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c58a9294-30c7-4e10-b71d-c88ec7dae0f8 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:13.785010 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c58a9294-30c7-4e10-b71d-c88ec7dae0f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:13.785312 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c58a9294-30c7-4e10-b71d-c88ec7dae0f8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:13.786576 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c58a9294-30c7-4e10-b71d-c88ec7dae0f8 None admin] RBAC: Adding query filter params (role.id=414286927fdd44688107ff5e10235acf, scope.project.id=69bfdc060ff949e2ba795cad23f9f746, user.id=04afced9034d4384ada854ebde4960f5) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:13.786884 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c58a9294-30c7-4e10-b71d-c88ec7dae0f8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:13.788355 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c58a9294-30c7-4e10-b71d-c88ec7dae0f8 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:13.795665 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 94/186] 10.222.0.42 () {56 vars in 1374 bytes} [Thu Oct 27 01:11:13 2022] GET /identity/v3/role_assignments?user.id=04afced9034d4384ada854ebde4960f5&scope.project.id=69bfdc060ff949e2ba795cad23f9f746&role.id=414286927fdd44688107ff5e10235acf => generated 608 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:11:14.758551 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af9fb6b-6d02-4036-8b91-0b5c98a731ca None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:14.759082 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af9fb6b-6d02-4036-8b91-0b5c98a731ca None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:14.759082 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af9fb6b-6d02-4036-8b91-0b5c98a731ca None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:14.760469 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 93/187] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:14.766561 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a49707b-e82b-4754-a094-0bfa8d45f7f0 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:14.767099 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a49707b-e82b-4754-a094-0bfa8d45f7f0 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:14.767099 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a49707b-e82b-4754-a094-0bfa8d45f7f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:14.787018 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-2a49707b-e82b-4754-a094-0bfa8d45f7f0 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:14.788253 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-2a49707b-e82b-4754-a094-0bfa8d45f7f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:14.857984 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 95/188] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:11:14.883197 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c818e597-bb57-4ecf-a9cf-426a62bd3b39 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:14.883491 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c818e597-bb57-4ecf-a9cf-426a62bd3b39 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:14.883758 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c818e597-bb57-4ecf-a9cf-426a62bd3b39 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:14.885256 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 94/189] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:14.890195 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be25077f-54a9-4014-9a3e-37e9808fca90 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:14.890480 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be25077f-54a9-4014-9a3e-37e9808fca90 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:14.890756 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be25077f-54a9-4014-9a3e-37e9808fca90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:14.908801 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-be25077f-54a9-4014-9a3e-37e9808fca90 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:14.910002 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-be25077f-54a9-4014-9a3e-37e9808fca90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:14.939834 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:14.978747 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 96/190] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:11:15.125578 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c818e597-bb57-4ecf-a9cf-426a62bd3b39 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:15.184273 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:15.227032 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c818e597-bb57-4ecf-a9cf-426a62bd3b39 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:15.247611 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be25077f-54a9-4014-9a3e-37e9808fca90 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:15.261351 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df17325-8dae-4605-969f-d7c65805e21f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:15.261980 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df17325-8dae-4605-969f-d7c65805e21f None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:15.262257 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df17325-8dae-4605-969f-d7c65805e21f None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:15.262527 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df17325-8dae-4605-969f-d7c65805e21f None admin] PATH_INFO: `/v3/domains/default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:15.263616 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9df17325-8dae-4605-969f-d7c65805e21f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:15.265096 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9df17325-8dae-4605-969f-d7c65805e21f None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:15.269545 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 95/191] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:11:15 2022] GET /identity/v3/domains/default => generated 199 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:15.272820 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df17325-8dae-4605-969f-d7c65805e21f None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:15.355681 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be25077f-54a9-4014-9a3e-37e9808fca90 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:15.377262 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df17325-8dae-4605-969f-d7c65805e21f None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:15.393823 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9bafb4a-0670-4706-9ab1-052119af9eb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:15.394475 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bafb4a-0670-4706-9ab1-052119af9eb8 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:15.394946 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bafb4a-0670-4706-9ab1-052119af9eb8 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:15.395050 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bafb4a-0670-4706-9ab1-052119af9eb8 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:15.405911 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:15.409151 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9bafb4a-0670-4706-9ab1-052119af9eb8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:15.414790 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988ed747-0f81-46b6-ab17-6e7699909951 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:15.415405 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988ed747-0f81-46b6-ab17-6e7699909951 None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:15.415504 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-e9bafb4a-0670-4706-9ab1-052119af9eb8 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 27 01:11:15.415504 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:15.415504 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:15.415504 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:15.415504 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:15.415504 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:15.415504 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:15.415504 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:15.415504 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:15.415504 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:15.415504 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:15.415504 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:15.415504 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:15.415504 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:15.415504 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:15.415504 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:15.416330 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:15.416330 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:15.416330 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:15.416330 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:15.416330 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:15.416330 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:15.416330 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:15.416330 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:15.416330 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:15.416330 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:15.416330 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:15.416330 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:15.416330 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:15.416330 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:15.416330 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:15.416330 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:15.417294 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:15.417294 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:15.417294 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:15.417294 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:15.417294 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:15.417294 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:15.417294 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:15.417294 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:15.417294 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:15.417294 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:15.417294 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 27 01:11:15.417294 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:15.418079 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988ed747-0f81-46b6-ab17-6e7699909951 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:15.418079 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988ed747-0f81-46b6-ab17-6e7699909951 None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:15.418079 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988ed747-0f81-46b6-ab17-6e7699909951 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:15.418247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 97/192] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:11:15 2022] GET /identity/v3/roles/reader => generated 84 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 27 01:11:15.422765 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9bafb4a-0670-4706-9ab1-052119af9eb8 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:15.423355 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988ed747-0f81-46b6-ab17-6e7699909951 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:15.448699 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 96/193] 10.222.0.42 () {60 vars in 1132 bytes} [Thu Oct 27 01:11:15 2022] POST /identity/v3/users => generated 293 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:11:15.462264 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:15.525295 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62b56ba-c0e5-4032-9a4d-62e3b90c1ddb None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:15.526176 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62b56ba-c0e5-4032-9a4d-62e3b90c1ddb None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:15.526521 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62b56ba-c0e5-4032-9a4d-62e3b90c1ddb None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:15.528167 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 97/194] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 27 01:11:15.534043 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd313e5-f94b-461f-b4fd-483695d30bd1 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:15.534478 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd313e5-f94b-461f-b4fd-483695d30bd1 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:15.534848 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd313e5-f94b-461f-b4fd-483695d30bd1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:15.549829 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9bafb4a-0670-4706-9ab1-052119af9eb8 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:15.554961 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-6cd313e5-f94b-461f-b4fd-483695d30bd1 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:15.556556 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-6cd313e5-f94b-461f-b4fd-483695d30bd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:15.589505 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13de069a-957f-43b8-a3eb-9dc179a403d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:15.590485 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13de069a-957f-43b8-a3eb-9dc179a403d9 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:15.591048 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13de069a-957f-43b8-a3eb-9dc179a403d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:15.591326 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13de069a-957f-43b8-a3eb-9dc179a403d9 None admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:15.592025 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13de069a-957f-43b8-a3eb-9dc179a403d9 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:15.592321 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13de069a-957f-43b8-a3eb-9dc179a403d9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:15.593877 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13de069a-957f-43b8-a3eb-9dc179a403d9 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:15.605778 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 98/195] 10.222.0.42 () {56 vars in 1101 bytes} [Thu Oct 27 01:11:15 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:11:15.612071 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13de069a-957f-43b8-a3eb-9dc179a403d9 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:15.641268 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 98/196] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 27 01:11:15.714155 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:15.728154 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13de069a-957f-43b8-a3eb-9dc179a403d9 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:15.766276 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12133912-6158-4f2f-8484-ff1d823d6923 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:15.766915 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12133912-6158-4f2f-8484-ff1d823d6923 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:15.767426 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12133912-6158-4f2f-8484-ff1d823d6923 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:15.767426 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12133912-6158-4f2f-8484-ff1d823d6923 None admin] PATH_INFO: `/v3/users/6008326c6e544835b08a339a0a80def6` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:15.783022 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12133912-6158-4f2f-8484-ff1d823d6923 None admin] RBAC: Authorizing `identity:get_user(user_id=6008326c6e544835b08a339a0a80def6)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:15.784864 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12133912-6158-4f2f-8484-ff1d823d6923 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:15.787483 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 99/197] 10.222.0.42 () {56 vars in 1177 bytes} [Thu Oct 27 01:11:15 2022] GET /identity/v3/users/6008326c6e544835b08a339a0a80def6 => generated 297 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:15.793231 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd313e5-f94b-461f-b4fd-483695d30bd1 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:15.903943 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd313e5-f94b-461f-b4fd-483695d30bd1 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:15.911444 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12133912-6158-4f2f-8484-ff1d823d6923 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:15.922556 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:15.958745 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915d698d-5b9e-4b1c-85ba-1a1486dee0f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:15.959386 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915d698d-5b9e-4b1c-85ba-1a1486dee0f7 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:15.959659 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915d698d-5b9e-4b1c-85ba-1a1486dee0f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:15.959927 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915d698d-5b9e-4b1c-85ba-1a1486dee0f7 None admin] PATH_INFO: `/v3/projects/69bfdc060ff949e2ba795cad23f9f746` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:15.961650 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-915d698d-5b9e-4b1c-85ba-1a1486dee0f7 None admin] RBAC: Authorizing `identity:get_project(project_id=69bfdc060ff949e2ba795cad23f9f746)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:15.963115 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-915d698d-5b9e-4b1c-85ba-1a1486dee0f7 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:15.965734 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 99/198] 10.222.0.42 () {56 vars in 1186 bytes} [Thu Oct 27 01:11:15 2022] GET /identity/v3/projects/69bfdc060ff949e2ba795cad23f9f746 => generated 302 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:15.971717 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915d698d-5b9e-4b1c-85ba-1a1486dee0f7 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:16.035586 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12133912-6158-4f2f-8484-ff1d823d6923 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:16.088457 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514ee737-b087-4634-9ecd-cc9bd239b585 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:16.089107 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514ee737-b087-4634-9ecd-cc9bd239b585 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:16.089381 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514ee737-b087-4634-9ecd-cc9bd239b585 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:16.089680 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514ee737-b087-4634-9ecd-cc9bd239b585 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:16.095992 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915d698d-5b9e-4b1c-85ba-1a1486dee0f7 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:16.100811 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-514ee737-b087-4634-9ecd-cc9bd239b585 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:16.106007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-514ee737-b087-4634-9ecd-cc9bd239b585 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 27 01:11:16.106007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:16.106007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:16.106007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:16.106007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:16.106007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:16.106007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:16.106007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:16.106007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:16.106007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:16.106007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:16.106007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:16.106007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:16.106007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:16.106007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:16.106007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:16.106919 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:16.106919 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:16.106919 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:16.106919 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:16.106919 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:16.106919 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:16.106919 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:16.106919 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:16.106919 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:16.106919 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:16.106919 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:16.106919 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:16.106919 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:16.106919 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:16.106919 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:16.106919 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:16.107714 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:16.107714 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:16.107714 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:16.107714 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:16.107714 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:16.107714 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:16.107714 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:16.107714 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:16.107714 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:16.107714 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:16.107714 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 27 01:11:16.107714 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:16.111987 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 100/199] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:11:15 2022] GET /identity/v3/roles/member => generated 84 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 27 01:11:16.112862 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514ee737-b087-4634-9ecd-cc9bd239b585 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:16.125439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:16.143036 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37fb6e6-5e04-4e73-aca9-a57d9b02a16f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:16.144457 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37fb6e6-5e04-4e73-aca9-a57d9b02a16f None admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:16.145032 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37fb6e6-5e04-4e73-aca9-a57d9b02a16f None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:16.146215 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37fb6e6-5e04-4e73-aca9-a57d9b02a16f None admin] PATH_INFO: `/v3/projects/69bfdc060ff949e2ba795cad23f9f746/users/6008326c6e544835b08a339a0a80def6/roles/5fa0c579f1fb428e9dcae929dd21846f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:16.149118 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b37fb6e6-5e04-4e73-aca9-a57d9b02a16f None admin] RBAC: Authorizing `identity:create_grant(project_id=69bfdc060ff949e2ba795cad23f9f746, user_id=6008326c6e544835b08a339a0a80def6, role_id=5fa0c579f1fb428e9dcae929dd21846f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:16.151664 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b37fb6e6-5e04-4e73-aca9-a57d9b02a16f None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:16.170597 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 100/200] 10.222.0.42 () {58 vars in 1439 bytes} [Thu Oct 27 01:11:15 2022] PUT /identity/v3/projects/69bfdc060ff949e2ba795cad23f9f746/users/6008326c6e544835b08a339a0a80def6/roles/5fa0c579f1fb428e9dcae929dd21846f => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:11:16.240204 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514ee737-b087-4634-9ecd-cc9bd239b585 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:16.280426 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16339f7d-f6a5-4305-8fe9-41584be00964 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:16.281392 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16339f7d-f6a5-4305-8fe9-41584be00964 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:16.281804 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16339f7d-f6a5-4305-8fe9-41584be00964 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:16.282208 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16339f7d-f6a5-4305-8fe9-41584be00964 None admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:16.283016 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16339f7d-f6a5-4305-8fe9-41584be00964 None admin] RBAC: Adding query filter params (name=member) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:16.283453 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16339f7d-f6a5-4305-8fe9-41584be00964 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:16.285019 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16339f7d-f6a5-4305-8fe9-41584be00964 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:16.295013 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 101/201] 10.222.0.42 () {56 vars in 1101 bytes} [Thu Oct 27 01:11:16 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:16.301176 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37fb6e6-5e04-4e73-aca9-a57d9b02a16f None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:16.351480 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:16.418368 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37fb6e6-5e04-4e73-aca9-a57d9b02a16f None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:16.454703 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6ad4a0-c18f-4dc7-889c-af79ea14d39a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:16.455558 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6ad4a0-c18f-4dc7-889c-af79ea14d39a None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:16.455975 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6ad4a0-c18f-4dc7-889c-af79ea14d39a None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:16.456384 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6ad4a0-c18f-4dc7-889c-af79ea14d39a None admin] PATH_INFO: `/v3/users/c40599d01dd34fb8a1bf426d02fd12e2` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:16.457659 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a6ad4a0-c18f-4dc7-889c-af79ea14d39a None admin] RBAC: Authorizing `identity:get_user(user_id=c40599d01dd34fb8a1bf426d02fd12e2)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:16.460089 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a6ad4a0-c18f-4dc7-889c-af79ea14d39a None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:16.461873 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 101/202] 10.222.0.42 () {56 vars in 1177 bytes} [Thu Oct 27 01:11:16 2022] GET /identity/v3/users/c40599d01dd34fb8a1bf426d02fd12e2 => generated 297 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:16.468162 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16339f7d-f6a5-4305-8fe9-41584be00964 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:16.540142 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:16.582502 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16339f7d-f6a5-4305-8fe9-41584be00964 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:16.620641 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b84618d-dfa4-4240-b6bf-d9d96066c650 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:16.621697 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b84618d-dfa4-4240-b6bf-d9d96066c650 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:16.622238 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b84618d-dfa4-4240-b6bf-d9d96066c650 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:16.622723 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b84618d-dfa4-4240-b6bf-d9d96066c650 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:16.697117 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b84618d-dfa4-4240-b6bf-d9d96066c650 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:16.697117 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b84618d-dfa4-4240-b6bf-d9d96066c650 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:16.763340 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:16.798879 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 102/203] 10.222.0.42 () {58 vars in 1282 bytes} [Thu Oct 27 01:11:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 27 01:11:16.814949 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6ad4a0-c18f-4dc7-889c-af79ea14d39a None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:16.924878 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6ad4a0-c18f-4dc7-889c-af79ea14d39a None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:16.935686 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:16.962538 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d7afca-9be2-4f1a-9c24-e0fa755cdc3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:16.963287 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d7afca-9be2-4f1a-9c24-e0fa755cdc3b None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:16.963568 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d7afca-9be2-4f1a-9c24-e0fa755cdc3b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:16.963832 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d7afca-9be2-4f1a-9c24-e0fa755cdc3b None admin] PATH_INFO: `/v3/projects/69bfdc060ff949e2ba795cad23f9f746` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:16.964958 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d7afca-9be2-4f1a-9c24-e0fa755cdc3b None admin] RBAC: Authorizing `identity:get_project(project_id=69bfdc060ff949e2ba795cad23f9f746)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:16.966437 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d7afca-9be2-4f1a-9c24-e0fa755cdc3b None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:16.968232 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 102/204] 10.222.0.42 () {56 vars in 1186 bytes} [Thu Oct 27 01:11:16 2022] GET /identity/v3/projects/69bfdc060ff949e2ba795cad23f9f746 => generated 302 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:16.974513 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b84618d-dfa4-4240-b6bf-d9d96066c650 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:17.076315 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b84618d-dfa4-4240-b6bf-d9d96066c650 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:17.111204 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e77a8c-17b9-43de-b718-b3a417781330 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:17.111805 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e77a8c-17b9-43de-b718-b3a417781330 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:17.112078 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e77a8c-17b9-43de-b718-b3a417781330 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:17.112346 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e77a8c-17b9-43de-b718-b3a417781330 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:17.113080 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e77a8c-17b9-43de-b718-b3a417781330 None admin] RBAC: Adding query filter params (role.id=f199b79a7178462fa67bb8243943408f, scope.project.id=69bfdc060ff949e2ba795cad23f9f746, user.id=c40599d01dd34fb8a1bf426d02fd12e2) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:17.113382 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e77a8c-17b9-43de-b718-b3a417781330 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:17.114950 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e77a8c-17b9-43de-b718-b3a417781330 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:17.121908 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 103/205] 10.222.0.42 () {56 vars in 1374 bytes} [Thu Oct 27 01:11:16 2022] GET /identity/v3/role_assignments?user.id=c40599d01dd34fb8a1bf426d02fd12e2&scope.project.id=69bfdc060ff949e2ba795cad23f9f746&role.id=f199b79a7178462fa67bb8243943408f => generated 608 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:17.836083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef607e3f-9f27-4a97-8f9c-8dbb038c7bf5 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:17.836604 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef607e3f-9f27-4a97-8f9c-8dbb038c7bf5 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:17.836604 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef607e3f-9f27-4a97-8f9c-8dbb038c7bf5 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:17.838175 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 103/206] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:17.843154 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8954af7-3377-4985-820e-7d63dab3ae06 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:17.843524 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8954af7-3377-4985-820e-7d63dab3ae06 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:17.843706 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8954af7-3377-4985-820e-7d63dab3ae06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:17.862244 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-e8954af7-3377-4985-820e-7d63dab3ae06 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:17.863439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-e8954af7-3377-4985-820e-7d63dab3ae06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:17.891212 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:17.896606 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4bb2eec-5f1f-4c6b-a1c4-d57378df67df None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:17.896888 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4bb2eec-5f1f-4c6b-a1c4-d57378df67df None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:17.897211 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4bb2eec-5f1f-4c6b-a1c4-d57378df67df None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:17.898815 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 104/207] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 27 01:11:17.903706 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46acb8cf-10ec-416d-a18d-cab650a2462d None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:17.904200 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46acb8cf-10ec-416d-a18d-cab650a2462d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:17.904627 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46acb8cf-10ec-416d-a18d-cab650a2462d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:17.923972 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-46acb8cf-10ec-416d-a18d-cab650a2462d None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:17.925197 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-46acb8cf-10ec-416d-a18d-cab650a2462d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:17.937773 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 104/208] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:11:17.993279 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 105/209] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 27 01:11:18.205647 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8954af7-3377-4985-820e-7d63dab3ae06 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:18.261580 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46acb8cf-10ec-416d-a18d-cab650a2462d None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:18.308801 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8954af7-3377-4985-820e-7d63dab3ae06 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:18.345830 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58b66ed-ab04-484e-8f27-46f7a21325c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:18.346530 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58b66ed-ab04-484e-8f27-46f7a21325c4 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:18.346811 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58b66ed-ab04-484e-8f27-46f7a21325c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:18.347116 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58b66ed-ab04-484e-8f27-46f7a21325c4 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:18.358956 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58b66ed-ab04-484e-8f27-46f7a21325c4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:18.363860 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-d58b66ed-ab04-484e-8f27-46f7a21325c4 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 27 01:11:18.363860 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:18.363860 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:18.363860 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:18.363860 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:18.363860 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:18.363860 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:18.363860 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:18.363860 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:18.363860 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:18.363860 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:18.363860 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:18.363860 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:18.363860 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:18.363860 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:18.363860 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:18.365335 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:18.365335 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:18.365335 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:18.365335 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:18.365335 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:18.365335 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:18.365335 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:18.365335 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:18.365335 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:18.365335 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:18.365335 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:18.365335 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:18.365335 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:18.365335 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:18.365335 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:18.365335 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:18.366283 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:18.366283 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:18.366283 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:18.366283 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:18.366283 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:18.366283 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:18.366283 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:18.366283 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:18.366283 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:18.366283 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:18.366283 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 27 01:11:18.366283 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:18.366283 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 105/210] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:11:18 2022] GET /identity/v3/roles/member => generated 84 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:18.372820 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58b66ed-ab04-484e-8f27-46f7a21325c4 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:18.374969 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46acb8cf-10ec-416d-a18d-cab650a2462d None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:18.383057 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:18.398106 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:18.414487 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37ecb89-9f7f-4cf5-9295-b668a8b797c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:18.415211 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37ecb89-9f7f-4cf5-9295-b668a8b797c6 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:18.415501 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37ecb89-9f7f-4cf5-9295-b668a8b797c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:18.415754 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37ecb89-9f7f-4cf5-9295-b668a8b797c6 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:18.416904 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b37ecb89-9f7f-4cf5-9295-b668a8b797c6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:18.418417 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b37ecb89-9f7f-4cf5-9295-b668a8b797c6 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:18.421099 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 106/211] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:11:18 2022] GET /identity/v3/domains/default => generated 199 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:11:18.425831 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37ecb89-9f7f-4cf5-9295-b668a8b797c6 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:18.477019 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58b66ed-ab04-484e-8f27-46f7a21325c4 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:18.510005 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33187876-5498-458c-af23-081ad1d56eb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:18.510598 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33187876-5498-458c-af23-081ad1d56eb7 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:18.510868 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33187876-5498-458c-af23-081ad1d56eb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:18.511131 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33187876-5498-458c-af23-081ad1d56eb7 None admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:18.511783 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33187876-5498-458c-af23-081ad1d56eb7 None admin] RBAC: Adding query filter params (name=member) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:18.512075 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33187876-5498-458c-af23-081ad1d56eb7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:18.513470 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33187876-5498-458c-af23-081ad1d56eb7 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:18.521978 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 106/212] 10.222.0.42 () {56 vars in 1101 bytes} [Thu Oct 27 01:11:18 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:11:18.523103 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c306d4-b386-4ee6-aedf-1d8497a985d1 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:18.523383 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c306d4-b386-4ee6-aedf-1d8497a985d1 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:18.523652 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c306d4-b386-4ee6-aedf-1d8497a985d1 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:18.524823 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 107/213] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:18 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:18.527353 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c306d4-b386-4ee6-aedf-1d8497a985d1 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:18.530295 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37ecb89-9f7f-4cf5-9295-b668a8b797c6 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:18.564518 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1280e0be-fe16-44d7-9a34-bfa7ed835add None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:18.565152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1280e0be-fe16-44d7-9a34-bfa7ed835add None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:18.565435 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1280e0be-fe16-44d7-9a34-bfa7ed835add None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:18.565707 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1280e0be-fe16-44d7-9a34-bfa7ed835add None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:18.566386 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1280e0be-fe16-44d7-9a34-bfa7ed835add None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:18.567916 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1280e0be-fe16-44d7-9a34-bfa7ed835add None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:18.589999 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 107/214] 10.222.0.42 () {60 vars in 1132 bytes} [Thu Oct 27 01:11:18 2022] POST /identity/v3/users => generated 293 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:11:18.591202 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25526025-3a6c-4c2f-8450-7ba8f59fab9e None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:18.591489 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25526025-3a6c-4c2f-8450-7ba8f59fab9e None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:18.591761 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25526025-3a6c-4c2f-8450-7ba8f59fab9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:18.611902 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-25526025-3a6c-4c2f-8450-7ba8f59fab9e None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:18.613175 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-25526025-3a6c-4c2f-8450-7ba8f59fab9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:18.621974 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:18.641764 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:18.643213 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c306d4-b386-4ee6-aedf-1d8497a985d1 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:18.683251 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7844a4bc-6ed3-4a0f-affb-b16d496594f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:18.683910 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7844a4bc-6ed3-4a0f-affb-b16d496594f9 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:18.684192 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7844a4bc-6ed3-4a0f-affb-b16d496594f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:18.684468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7844a4bc-6ed3-4a0f-affb-b16d496594f9 None admin] PATH_INFO: `/v3/users/549341ef566b4758821e81a97b2f848e` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:18.694960 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 108/215] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:11:18.699965 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7844a4bc-6ed3-4a0f-affb-b16d496594f9 None admin] RBAC: Authorizing `identity:get_user(user_id=549341ef566b4758821e81a97b2f848e)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:18.701497 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7844a4bc-6ed3-4a0f-affb-b16d496594f9 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:18.703521 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 108/216] 10.222.0.42 () {56 vars in 1177 bytes} [Thu Oct 27 01:11:18 2022] GET /identity/v3/users/549341ef566b4758821e81a97b2f848e => generated 293 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:11:18.709584 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25526025-3a6c-4c2f-8450-7ba8f59fab9e None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:18.821900 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25526025-3a6c-4c2f-8450-7ba8f59fab9e None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:18.860217 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f91671c-2fb5-4da4-bd00-b8f7c6b20530 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:18.861549 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f91671c-2fb5-4da4-bd00-b8f7c6b20530 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:18.861966 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f91671c-2fb5-4da4-bd00-b8f7c6b20530 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:18.862373 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f91671c-2fb5-4da4-bd00-b8f7c6b20530 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:18.863252 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f91671c-2fb5-4da4-bd00-b8f7c6b20530 None admin] RBAC: Adding query filter params (role.id=f199b79a7178462fa67bb8243943408f, user.id=549341ef566b4758821e81a97b2f848e, scope.system=all) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:18.863695 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f91671c-2fb5-4da4-bd00-b8f7c6b20530 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:18.866563 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f91671c-2fb5-4da4-bd00-b8f7c6b20530 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:18.871985 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 109/217] 10.222.0.42 () {56 vars in 1308 bytes} [Thu Oct 27 01:11:18 2022] GET /identity/v3/role_assignments?user.id=549341ef566b4758821e81a97b2f848e&scope.system=all&role.id=f199b79a7178462fa67bb8243943408f => generated 226 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:18.965168 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7844a4bc-6ed3-4a0f-affb-b16d496594f9 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:19.084434 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7844a4bc-6ed3-4a0f-affb-b16d496594f9 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:19.121826 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49da5c17-2cae-4c7b-b19b-2fa155232662 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:19.122706 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49da5c17-2cae-4c7b-b19b-2fa155232662 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:19.123118 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49da5c17-2cae-4c7b-b19b-2fa155232662 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:19.123525 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49da5c17-2cae-4c7b-b19b-2fa155232662 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:19.134672 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49da5c17-2cae-4c7b-b19b-2fa155232662 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:19.139644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-49da5c17-2cae-4c7b-b19b-2fa155232662 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 27 01:11:19.139644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:19.139644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:19.139644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:19.139644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:19.139644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:19.139644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:19.139644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:19.139644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:19.139644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:19.139644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:19.139644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:19.139644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:19.139644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:19.139644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:19.139644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:19.140707 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:19.140707 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:19.140707 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:19.140707 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:19.140707 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:19.140707 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:19.140707 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:19.140707 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:19.140707 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:19.140707 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:19.140707 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:19.140707 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:19.140707 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:19.140707 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:19.140707 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:19.140707 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:19.141737 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:19.141737 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:19.141737 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:19.141737 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:19.141737 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:19.141737 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:19.141737 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:19.141737 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:19.141737 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:19.141737 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:19.141737 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 27 01:11:19.141737 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:19.143686 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 109/218] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:11:18 2022] GET /identity/v3/roles/reader => generated 84 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:19.150149 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f91671c-2fb5-4da4-bd00-b8f7c6b20530 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:19.179661 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:19.259707 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f91671c-2fb5-4da4-bd00-b8f7c6b20530 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:19.295085 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb7ffb8-a7bb-4fcd-8da5-6198f86342f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:19.295694 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb7ffb8-a7bb-4fcd-8da5-6198f86342f9 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:19.295971 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb7ffb8-a7bb-4fcd-8da5-6198f86342f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:19.296241 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb7ffb8-a7bb-4fcd-8da5-6198f86342f9 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:19.296903 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb7ffb8-a7bb-4fcd-8da5-6198f86342f9 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:19.297263 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb7ffb8-a7bb-4fcd-8da5-6198f86342f9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:19.298629 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb7ffb8-a7bb-4fcd-8da5-6198f86342f9 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:19.308216 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 110/219] 10.222.0.42 () {56 vars in 1101 bytes} [Thu Oct 27 01:11:19 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:19.313989 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49da5c17-2cae-4c7b-b19b-2fa155232662 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:19.324697 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:19.421988 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49da5c17-2cae-4c7b-b19b-2fa155232662 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:19.462172 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4377018-3938-4134-a97c-e9711173cfe9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:19.463101 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4377018-3938-4134-a97c-e9711173cfe9 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:19.463553 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4377018-3938-4134-a97c-e9711173cfe9 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:19.464019 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4377018-3938-4134-a97c-e9711173cfe9 None admin] PATH_INFO: `/v3/users/6008326c6e544835b08a339a0a80def6` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:19.465358 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4377018-3938-4134-a97c-e9711173cfe9 None admin] RBAC: Authorizing `identity:get_user(user_id=6008326c6e544835b08a339a0a80def6)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:19.466990 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4377018-3938-4134-a97c-e9711173cfe9 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:19.469093 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 110/220] 10.222.0.42 () {56 vars in 1177 bytes} [Thu Oct 27 01:11:19 2022] GET /identity/v3/users/6008326c6e544835b08a339a0a80def6 => generated 297 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:19.481090 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb7ffb8-a7bb-4fcd-8da5-6198f86342f9 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:19.580165 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb7ffb8-a7bb-4fcd-8da5-6198f86342f9 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:19.581818 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:19.617373 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2764a1-2b68-4fe9-92a3-8dffbb3e794e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:19.617984 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2764a1-2b68-4fe9-92a3-8dffbb3e794e None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:19.618265 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2764a1-2b68-4fe9-92a3-8dffbb3e794e None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:19.618535 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2764a1-2b68-4fe9-92a3-8dffbb3e794e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:19.696156 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe2764a1-2b68-4fe9-92a3-8dffbb3e794e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:19.697594 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe2764a1-2b68-4fe9-92a3-8dffbb3e794e None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:19.786536 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:19.799424 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 111/221] 10.222.0.42 () {58 vars in 1282 bytes} [Thu Oct 27 01:11:19 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 27 01:11:19.805416 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4377018-3938-4134-a97c-e9711173cfe9 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:19.880410 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:19.913815 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4377018-3938-4134-a97c-e9711173cfe9 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:19.950213 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb1bbc51-6113-41c3-beeb-2a900b200cbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:19.951049 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1bbc51-6113-41c3-beeb-2a900b200cbf None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:19.951488 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1bbc51-6113-41c3-beeb-2a900b200cbf None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:19.951899 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1bbc51-6113-41c3-beeb-2a900b200cbf None admin] PATH_INFO: `/v3/projects/69bfdc060ff949e2ba795cad23f9f746` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:19.953218 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb1bbc51-6113-41c3-beeb-2a900b200cbf None admin] RBAC: Authorizing `identity:get_project(project_id=69bfdc060ff949e2ba795cad23f9f746)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:19.955140 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb1bbc51-6113-41c3-beeb-2a900b200cbf None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:19.957266 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 111/222] 10.222.0.42 () {56 vars in 1186 bytes} [Thu Oct 27 01:11:19 2022] GET /identity/v3/projects/69bfdc060ff949e2ba795cad23f9f746 => generated 302 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:19.963770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2764a1-2b68-4fe9-92a3-8dffbb3e794e None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:20.067296 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2764a1-2b68-4fe9-92a3-8dffbb3e794e None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:20.102468 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c82189-9db9-45d3-b64f-12d431fc87a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:20.103491 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c82189-9db9-45d3-b64f-12d431fc87a9 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:20.103938 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c82189-9db9-45d3-b64f-12d431fc87a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:20.104367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c82189-9db9-45d3-b64f-12d431fc87a9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:20.105272 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c82189-9db9-45d3-b64f-12d431fc87a9 None admin] RBAC: Adding query filter params (role.id=5fa0c579f1fb428e9dcae929dd21846f, scope.project.id=69bfdc060ff949e2ba795cad23f9f746, user.id=6008326c6e544835b08a339a0a80def6) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:20.105738 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c82189-9db9-45d3-b64f-12d431fc87a9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:20.107321 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c82189-9db9-45d3-b64f-12d431fc87a9 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:20.114460 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 112/223] 10.222.0.42 () {56 vars in 1374 bytes} [Thu Oct 27 01:11:19 2022] GET /identity/v3/role_assignments?user.id=6008326c6e544835b08a339a0a80def6&scope.project.id=69bfdc060ff949e2ba795cad23f9f746&role.id=5fa0c579f1fb428e9dcae929dd21846f => generated 608 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:20.976225 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a97a9df-5b87-4837-8490-fc33e683b719 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:20.976724 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a97a9df-5b87-4837-8490-fc33e683b719 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:20.976796 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a97a9df-5b87-4837-8490-fc33e683b719 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:20.978189 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 112/224] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:20.983348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9415c4-0c7b-49e2-ad9c-3df28e64c188 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:20.983734 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9415c4-0c7b-49e2-ad9c-3df28e64c188 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:20.983906 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9415c4-0c7b-49e2-ad9c-3df28e64c188 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:20.992983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb91bd6-1a37-4704-bb38-c924860e85be None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:20.993277 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb91bd6-1a37-4704-bb38-c924860e85be None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:20.993555 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb91bd6-1a37-4704-bb38-c924860e85be None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:20.994909 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 113/225] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 27 01:11:20.999906 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9eeb0ce-5657-402b-a398-990587c42b46 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:21.000189 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9eeb0ce-5657-402b-a398-990587c42b46 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:21.000464 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9eeb0ce-5657-402b-a398-990587c42b46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:21.002305 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-cb9415c4-0c7b-49e2-ad9c-3df28e64c188 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:21.003482 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-cb9415c4-0c7b-49e2-ad9c-3df28e64c188 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:21.018556 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-a9eeb0ce-5657-402b-a398-990587c42b46 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:21.019737 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-a9eeb0ce-5657-402b-a398-990587c42b46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:21.059796 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:21.084083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 113/226] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:11:21.093768 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 114/227] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 27 01:11:21.186686 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365602e9-abbb-46b1-9163-ce1cd3c952cb None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:21.186991 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365602e9-abbb-46b1-9163-ce1cd3c952cb None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:21.187238 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365602e9-abbb-46b1-9163-ce1cd3c952cb None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:21.188734 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 114/228] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:21.194430 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08791377-4e41-40aa-bdf1-a255bbbd31f0 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:21.194737 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08791377-4e41-40aa-bdf1-a255bbbd31f0 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:21.194974 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08791377-4e41-40aa-bdf1-a255bbbd31f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:21.213095 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-08791377-4e41-40aa-bdf1-a255bbbd31f0 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:21.214361 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-08791377-4e41-40aa-bdf1-a255bbbd31f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:21.275969 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:21.287376 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 115/229] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 27 01:11:21.354288 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365602e9-abbb-46b1-9163-ce1cd3c952cb None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:21.366684 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08791377-4e41-40aa-bdf1-a255bbbd31f0 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:21.462314 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365602e9-abbb-46b1-9163-ce1cd3c952cb None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:21.476944 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08791377-4e41-40aa-bdf1-a255bbbd31f0 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:21.498466 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25515fba-972c-4ed3-ab27-a350c8cf2167 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:21.499094 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25515fba-972c-4ed3-ab27-a350c8cf2167 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:21.499376 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25515fba-972c-4ed3-ab27-a350c8cf2167 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:21.499652 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25515fba-972c-4ed3-ab27-a350c8cf2167 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:21.500757 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25515fba-972c-4ed3-ab27-a350c8cf2167 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:21.502250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25515fba-972c-4ed3-ab27-a350c8cf2167 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:21.503841 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 115/230] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:11:21 2022] GET /identity/v3/domains/default => generated 199 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:21.509231 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25515fba-972c-4ed3-ab27-a350c8cf2167 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:21.511914 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5c55a5-8d42-48d7-9c66-1131099eb745 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:21.513224 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5c55a5-8d42-48d7-9c66-1131099eb745 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:21.513224 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5c55a5-8d42-48d7-9c66-1131099eb745 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:21.513224 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5c55a5-8d42-48d7-9c66-1131099eb745 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:21.524076 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d5c55a5-8d42-48d7-9c66-1131099eb745 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:21.528837 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-5d5c55a5-8d42-48d7-9c66-1131099eb745 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 27 01:11:21.528837 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:21.528837 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:21.528837 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:21.528837 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:21.528837 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:21.528837 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:21.528837 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:21.528837 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:21.528837 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:21.528837 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:21.528837 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:21.528837 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:21.528837 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:21.528837 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:21.528837 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:21.529746 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:21.529746 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:21.529746 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:21.529746 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:21.529746 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:21.529746 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:21.529746 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:21.529746 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:21.529746 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:21.529746 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:21.529746 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:21.529746 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:21.529746 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:21.529746 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:21.529746 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:21.529746 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:21.530524 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:21.530524 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:21.530524 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:21.530524 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:21.530524 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:21.530524 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:21.530524 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:21.530524 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:21.530524 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:21.530524 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:21.530524 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 27 01:11:21.530524 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:21.530524 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 116/231] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:11:21 2022] GET /identity/v3/roles/reader => generated 84 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 27 01:11:21.535460 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5c55a5-8d42-48d7-9c66-1131099eb745 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:21.568043 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:21.594152 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:21.620110 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25515fba-972c-4ed3-ab27-a350c8cf2167 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:21.634293 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5c55a5-8d42-48d7-9c66-1131099eb745 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:21.653390 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67952ce-72ca-48e3-b5a4-aae9e183022d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:21.653996 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67952ce-72ca-48e3-b5a4-aae9e183022d None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:21.654274 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67952ce-72ca-48e3-b5a4-aae9e183022d None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:21.654546 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67952ce-72ca-48e3-b5a4-aae9e183022d None admin] PATH_INFO: `/v3/groups` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:21.655171 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67952ce-72ca-48e3-b5a4-aae9e183022d None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:21.656625 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67952ce-72ca-48e3-b5a4-aae9e183022d None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:21.669263 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-118c9c31-472e-4524-af45-6ed1bcc49c09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:21.669917 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118c9c31-472e-4524-af45-6ed1bcc49c09 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:21.670192 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118c9c31-472e-4524-af45-6ed1bcc49c09 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:21.670468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118c9c31-472e-4524-af45-6ed1bcc49c09 None admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:21.671140 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-118c9c31-472e-4524-af45-6ed1bcc49c09 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:21.671434 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-118c9c31-472e-4524-af45-6ed1bcc49c09 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:21.672806 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-118c9c31-472e-4524-af45-6ed1bcc49c09 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:21.675369 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 116/232] 10.222.0.42 () {60 vars in 1134 bytes} [Thu Oct 27 01:11:21 2022] POST /identity/v3/groups => generated 229 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:11:21.676171 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67952ce-72ca-48e3-b5a4-aae9e183022d None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:21.682680 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 117/233] 10.222.0.42 () {56 vars in 1101 bytes} [Thu Oct 27 01:11:21 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:11:21.688601 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-118c9c31-472e-4524-af45-6ed1bcc49c09 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:21.797198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67952ce-72ca-48e3-b5a4-aae9e183022d None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:21.810379 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:21.812183 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-118c9c31-472e-4524-af45-6ed1bcc49c09 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:21.836154 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:21.843062 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256c1f4f-c213-4028-be36-1942295425cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:21.844277 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256c1f4f-c213-4028-be36-1942295425cf None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:21.844555 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256c1f4f-c213-4028-be36-1942295425cf None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:21.844820 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256c1f4f-c213-4028-be36-1942295425cf None admin] PATH_INFO: `/v3/roles/member` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:21.851972 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52423319-3c9b-4acd-a3f7-25bf47e70ecd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:21.852147 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52423319-3c9b-4acd-a3f7-25bf47e70ecd None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:21.852428 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52423319-3c9b-4acd-a3f7-25bf47e70ecd None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:21.852702 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52423319-3c9b-4acd-a3f7-25bf47e70ecd None admin] PATH_INFO: `/v3/users/6cd9a9702f5b4db2b10e8e0de4f20bad` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:21.857594 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-256c1f4f-c213-4028-be36-1942295425cf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:21.863804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-256c1f4f-c213-4028-be36-1942295425cf None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 27 01:11:21.863804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:21.863804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:21.863804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:21.863804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:21.863804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:21.863804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:21.863804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:21.863804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:21.863804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:21.863804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:21.863804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:21.863804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:21.863804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:21.863804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:21.863804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:21.865050 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:21.865050 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:21.865050 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:21.865050 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:21.865050 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:21.865050 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:21.865050 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:21.865050 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:21.865050 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:21.865050 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:21.865050 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:21.865050 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:21.865050 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:21.865050 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:21.865050 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:21.865050 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:21.865899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:21.865899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:21.865899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:21.865899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:21.865899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:21.865899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:21.865899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:21.865899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:21.865899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:21.865899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:21.865899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 27 01:11:21.865899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:21.865899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 117/234] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:11:21 2022] GET /identity/v3/roles/member => generated 84 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:21.869304 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52423319-3c9b-4acd-a3f7-25bf47e70ecd None admin] RBAC: Authorizing `identity:get_user(user_id=6cd9a9702f5b4db2b10e8e0de4f20bad)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:21.870833 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52423319-3c9b-4acd-a3f7-25bf47e70ecd None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:21.872812 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256c1f4f-c213-4028-be36-1942295425cf None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:21.872928 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 118/235] 10.222.0.42 () {56 vars in 1177 bytes} [Thu Oct 27 01:11:21 2022] GET /identity/v3/users/6cd9a9702f5b4db2b10e8e0de4f20bad => generated 293 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:11:21.878156 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52423319-3c9b-4acd-a3f7-25bf47e70ecd None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:21.985293 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256c1f4f-c213-4028-be36-1942295425cf None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:21.995088 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52423319-3c9b-4acd-a3f7-25bf47e70ecd None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:22.022266 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd4a062-ec5a-4626-b2ca-97d6b8941025 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:22.023169 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd4a062-ec5a-4626-b2ca-97d6b8941025 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:22.023451 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd4a062-ec5a-4626-b2ca-97d6b8941025 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:22.023729 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd4a062-ec5a-4626-b2ca-97d6b8941025 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:22.024401 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd4a062-ec5a-4626-b2ca-97d6b8941025 None admin] RBAC: Adding query filter params (name=member) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:22.024706 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd4a062-ec5a-4626-b2ca-97d6b8941025 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:22.026138 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd4a062-ec5a-4626-b2ca-97d6b8941025 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:22.035097 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 118/236] 10.222.0.42 () {56 vars in 1101 bytes} [Thu Oct 27 01:11:21 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:11:22.036702 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5281727-628a-4931-bb4c-92055bede63d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:22.036948 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5281727-628a-4931-bb4c-92055bede63d None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:22.037256 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5281727-628a-4931-bb4c-92055bede63d None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:22.037529 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5281727-628a-4931-bb4c-92055bede63d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:22.038252 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5281727-628a-4931-bb4c-92055bede63d None admin] RBAC: Adding query filter params (role.id=5fa0c579f1fb428e9dcae929dd21846f, user.id=6cd9a9702f5b4db2b10e8e0de4f20bad, scope.system=all) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:22.038554 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5281727-628a-4931-bb4c-92055bede63d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:22.040063 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5281727-628a-4931-bb4c-92055bede63d None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:22.040995 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd4a062-ec5a-4626-b2ca-97d6b8941025 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:22.045562 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 119/237] 10.222.0.42 () {56 vars in 1308 bytes} [Thu Oct 27 01:11:21 2022] GET /identity/v3/role_assignments?user.id=6cd9a9702f5b4db2b10e8e0de4f20bad&scope.system=all&role.id=5fa0c579f1fb428e9dcae929dd21846f => generated 226 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:11:22.077996 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:22.158170 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd4a062-ec5a-4626-b2ca-97d6b8941025 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:22.197120 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f513df8f-378f-4b1e-af6a-ceb85b2d79b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:22.197784 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f513df8f-378f-4b1e-af6a-ceb85b2d79b6 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:22.198063 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f513df8f-378f-4b1e-af6a-ceb85b2d79b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:22.198334 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f513df8f-378f-4b1e-af6a-ceb85b2d79b6 None admin] PATH_INFO: `/v3/users/549341ef566b4758821e81a97b2f848e` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:22.199504 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f513df8f-378f-4b1e-af6a-ceb85b2d79b6 None admin] RBAC: Authorizing `identity:get_user(user_id=549341ef566b4758821e81a97b2f848e)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:22.200973 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f513df8f-378f-4b1e-af6a-ceb85b2d79b6 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:22.202831 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 119/238] 10.222.0.42 () {56 vars in 1177 bytes} [Thu Oct 27 01:11:22 2022] GET /identity/v3/users/549341ef566b4758821e81a97b2f848e => generated 293 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:11:22.208451 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5281727-628a-4931-bb4c-92055bede63d None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:22.256022 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:22.315408 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5281727-628a-4931-bb4c-92055bede63d None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:22.350384 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484aef9b-d027-4c3a-a5f4-357211f8d537 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:22.351041 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484aef9b-d027-4c3a-a5f4-357211f8d537 None admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:22.351319 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484aef9b-d027-4c3a-a5f4-357211f8d537 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:22.351591 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484aef9b-d027-4c3a-a5f4-357211f8d537 None admin] PATH_INFO: `/v3/system/users/549341ef566b4758821e81a97b2f848e/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:22.353047 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-484aef9b-d027-4c3a-a5f4-357211f8d537 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=549341ef566b4758821e81a97b2f848e, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:22.354455 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-484aef9b-d027-4c3a-a5f4-357211f8d537 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:22.363632 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 120/239] 10.222.0.42 () {58 vars in 1334 bytes} [Thu Oct 27 01:11:22 2022] PUT /identity/v3/system/users/549341ef566b4758821e81a97b2f848e/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:11:23.893189 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fdf3420-68a4-4f92-ba1c-ab42dae105f6 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:23.893691 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fdf3420-68a4-4f92-ba1c-ab42dae105f6 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:23.893763 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fdf3420-68a4-4f92-ba1c-ab42dae105f6 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:23.895298 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 120/240] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:23.900449 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613f7c85-5599-456b-b116-c2f5e775950c None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:23.900825 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613f7c85-5599-456b-b116-c2f5e775950c None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:23.901000 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613f7c85-5599-456b-b116-c2f5e775950c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:23.919685 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-613f7c85-5599-456b-b116-c2f5e775950c None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:23.920871 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-613f7c85-5599-456b-b116-c2f5e775950c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:23.989610 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 121/241] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:11:24.204101 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c177c88f-f62a-4c72-8add-61c3d0bba129 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:24.204483 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c177c88f-f62a-4c72-8add-61c3d0bba129 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:24.204656 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c177c88f-f62a-4c72-8add-61c3d0bba129 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:24.206074 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 121/242] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:24.211245 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746fe666-1d3c-4b92-8c85-cb20379ecda6 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:24.211528 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746fe666-1d3c-4b92-8c85-cb20379ecda6 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:24.211795 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746fe666-1d3c-4b92-8c85-cb20379ecda6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:24.230266 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-746fe666-1d3c-4b92-8c85-cb20379ecda6 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:24.231437 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-746fe666-1d3c-4b92-8c85-cb20379ecda6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:24.258998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c177c88f-f62a-4c72-8add-61c3d0bba129 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:24.261967 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:24.313088 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 122/243] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:11:24.360028 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:24.366647 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c177c88f-f62a-4c72-8add-61c3d0bba129 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:24.402878 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56dbf7d-2def-47b0-9b7a-3dcea04f4f76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:24.403535 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56dbf7d-2def-47b0-9b7a-3dcea04f4f76 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:24.403815 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56dbf7d-2def-47b0-9b7a-3dcea04f4f76 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:24.404087 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56dbf7d-2def-47b0-9b7a-3dcea04f4f76 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:24.405322 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f56dbf7d-2def-47b0-9b7a-3dcea04f4f76 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:24.406820 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f56dbf7d-2def-47b0-9b7a-3dcea04f4f76 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:24.408643 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 122/244] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:11:24 2022] GET /identity/v3/domains/default => generated 199 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:11:24.414988 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746fe666-1d3c-4b92-8c85-cb20379ecda6 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:24.507336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d444e3f-ab84-4753-a705-38ddfbf3153c None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:24.507634 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d444e3f-ab84-4753-a705-38ddfbf3153c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:24.508003 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d444e3f-ab84-4753-a705-38ddfbf3153c None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:24.509467 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 123/245] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 27 01:11:24.514234 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ca8aef-4606-41d2-9575-aea36867b5a8 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:24.514514 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ca8aef-4606-41d2-9575-aea36867b5a8 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:24.514786 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ca8aef-4606-41d2-9575-aea36867b5a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:24.523015 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746fe666-1d3c-4b92-8c85-cb20379ecda6 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:24.536812 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-a1ca8aef-4606-41d2-9575-aea36867b5a8 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:24.538275 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-a1ca8aef-4606-41d2-9575-aea36867b5a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:24.561568 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365fbc20-199d-428d-af72-1e2d42449a7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:24.562445 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365fbc20-199d-428d-af72-1e2d42449a7b None admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:24.563145 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365fbc20-199d-428d-af72-1e2d42449a7b None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:24.563424 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365fbc20-199d-428d-af72-1e2d42449a7b None admin] PATH_INFO: `/v3/groups` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:24.564348 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-365fbc20-199d-428d-af72-1e2d42449a7b None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:24.566462 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-365fbc20-199d-428d-af72-1e2d42449a7b None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:24.584564 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 123/246] 10.222.0.42 () {60 vars in 1134 bytes} [Thu Oct 27 01:11:24 2022] POST /identity/v3/groups => generated 226 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:11:24.585478 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365fbc20-199d-428d-af72-1e2d42449a7b None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:24.621176 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:24.621381 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 124/247] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 27 01:11:24.698376 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365fbc20-199d-428d-af72-1e2d42449a7b None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:24.735430 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098d1859-8853-4184-b867-9d944ee6017b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:24.736090 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098d1859-8853-4184-b867-9d944ee6017b None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:24.736370 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098d1859-8853-4184-b867-9d944ee6017b None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:24.736642 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098d1859-8853-4184-b867-9d944ee6017b None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:24.748225 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-098d1859-8853-4184-b867-9d944ee6017b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:24.753685 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-098d1859-8853-4184-b867-9d944ee6017b None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 27 01:11:24.753685 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:24.753685 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:24.753685 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:24.753685 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:24.753685 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:24.753685 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:24.753685 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:24.753685 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:24.753685 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:24.753685 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:24.753685 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:24.753685 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:24.753685 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:24.753685 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:24.753685 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:24.755092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:24.755092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:24.755092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:24.755092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:24.755092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:24.755092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:24.755092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:24.755092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:24.755092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:24.755092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:24.755092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:24.755092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:24.755092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:24.755092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:24.755092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:24.755092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:24.755864 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:24.755864 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:24.755864 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:24.755864 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:24.755864 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:24.755864 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:24.755864 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:24.755864 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:24.755864 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:24.755864 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:24.755864 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 27 01:11:24.755864 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:24.755864 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 124/248] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:11:24 2022] GET /identity/v3/roles/reader => generated 84 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:24.763085 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ca8aef-4606-41d2-9575-aea36867b5a8 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:24.863138 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:24.871131 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ca8aef-4606-41d2-9575-aea36867b5a8 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:24.885794 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098d1859-8853-4184-b867-9d944ee6017b None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:24.914864 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0922c8f4-8059-4219-97a2-b1d9472e6080 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:24.916162 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0922c8f4-8059-4219-97a2-b1d9472e6080 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:24.916437 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0922c8f4-8059-4219-97a2-b1d9472e6080 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:24.916705 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0922c8f4-8059-4219-97a2-b1d9472e6080 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:24.917425 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0922c8f4-8059-4219-97a2-b1d9472e6080 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:24.917727 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0922c8f4-8059-4219-97a2-b1d9472e6080 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:24.919294 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0922c8f4-8059-4219-97a2-b1d9472e6080 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:24.933156 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 125/249] 10.222.0.42 () {56 vars in 1101 bytes} [Thu Oct 27 01:11:24 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:24.938759 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0922c8f4-8059-4219-97a2-b1d9472e6080 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:24.988700 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:25.011315 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098d1859-8853-4184-b867-9d944ee6017b None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:25.060772 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e6cee5-0c0b-4ad1-8c20-a6e401cead6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:25.061363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e6cee5-0c0b-4ad1-8c20-a6e401cead6d None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:25.061637 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e6cee5-0c0b-4ad1-8c20-a6e401cead6d None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:25.061903 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e6cee5-0c0b-4ad1-8c20-a6e401cead6d None admin] PATH_INFO: `/v3/roles/member` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:25.062752 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0922c8f4-8059-4219-97a2-b1d9472e6080 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:25.075685 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e6cee5-0c0b-4ad1-8c20-a6e401cead6d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:25.083757 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-73e6cee5-0c0b-4ad1-8c20-a6e401cead6d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 27 01:11:25.083757 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:25.083757 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:25.083757 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:25.083757 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:25.083757 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:25.083757 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:25.083757 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:25.083757 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:25.083757 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:25.083757 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:25.083757 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:25.083757 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:25.083757 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:25.083757 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:25.083757 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:25.084669 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:25.084669 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:25.084669 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:25.084669 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:25.084669 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:25.084669 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:25.084669 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:25.084669 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:25.084669 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:25.084669 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:25.084669 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:25.084669 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:25.084669 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:25.084669 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:25.084669 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:25.084669 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:25.085507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:25.085507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:25.085507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:25.085507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:25.085507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:25.085507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:25.085507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:25.085507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:25.085507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:25.085507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:25.085507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 27 01:11:25.085507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:25.086107 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 125/250] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:11:24 2022] GET /identity/v3/roles/member => generated 84 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 27 01:11:25.091648 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e6cee5-0c0b-4ad1-8c20-a6e401cead6d None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:25.112235 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9003edec-a271-4f9d-bde9-1c7b5a1282a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:25.112931 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9003edec-a271-4f9d-bde9-1c7b5a1282a3 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:25.113536 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9003edec-a271-4f9d-bde9-1c7b5a1282a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:25.114469 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9003edec-a271-4f9d-bde9-1c7b5a1282a3 None admin] PATH_INFO: `/v3/users/6cd9a9702f5b4db2b10e8e0de4f20bad` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:25.116159 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9003edec-a271-4f9d-bde9-1c7b5a1282a3 None admin] RBAC: Authorizing `identity:get_user(user_id=6cd9a9702f5b4db2b10e8e0de4f20bad)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:25.117796 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9003edec-a271-4f9d-bde9-1c7b5a1282a3 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:25.120271 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 126/251] 10.222.0.42 () {56 vars in 1177 bytes} [Thu Oct 27 01:11:24 2022] GET /identity/v3/users/6cd9a9702f5b4db2b10e8e0de4f20bad => generated 293 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:11:25.126143 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9003edec-a271-4f9d-bde9-1c7b5a1282a3 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:25.161547 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:25.238322 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e6cee5-0c0b-4ad1-8c20-a6e401cead6d None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:25.274710 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9003edec-a271-4f9d-bde9-1c7b5a1282a3 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:25.290396 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc549eb5-e2c4-4f51-8ab7-9d36ea64059b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:25.291003 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc549eb5-e2c4-4f51-8ab7-9d36ea64059b None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:25.291276 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc549eb5-e2c4-4f51-8ab7-9d36ea64059b None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:25.291543 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc549eb5-e2c4-4f51-8ab7-9d36ea64059b None admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:25.292203 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc549eb5-e2c4-4f51-8ab7-9d36ea64059b None admin] RBAC: Adding query filter params (name=member) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:25.292499 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc549eb5-e2c4-4f51-8ab7-9d36ea64059b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:25.293919 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc549eb5-e2c4-4f51-8ab7-9d36ea64059b None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:25.303910 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 126/252] 10.222.0.42 () {56 vars in 1101 bytes} [Thu Oct 27 01:11:25 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:25.310287 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc549eb5-e2c4-4f51-8ab7-9d36ea64059b None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:25.313349 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc53b386-cedf-4d66-b459-8a1bb70b25b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:25.314069 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc53b386-cedf-4d66-b459-8a1bb70b25b2 None admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:25.314351 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc53b386-cedf-4d66-b459-8a1bb70b25b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:25.314624 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc53b386-cedf-4d66-b459-8a1bb70b25b2 None admin] PATH_INFO: `/v3/system/users/6cd9a9702f5b4db2b10e8e0de4f20bad/roles/5fa0c579f1fb428e9dcae929dd21846f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:25.316041 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc53b386-cedf-4d66-b459-8a1bb70b25b2 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=6cd9a9702f5b4db2b10e8e0de4f20bad, role_id=5fa0c579f1fb428e9dcae929dd21846f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:25.317699 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc53b386-cedf-4d66-b459-8a1bb70b25b2 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:25.323311 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:25.329312 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 127/253] 10.222.0.42 () {58 vars in 1334 bytes} [Thu Oct 27 01:11:25 2022] PUT /identity/v3/system/users/6cd9a9702f5b4db2b10e8e0de4f20bad/roles/5fa0c579f1fb428e9dcae929dd21846f => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:11:25.430772 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc549eb5-e2c4-4f51-8ab7-9d36ea64059b None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:25.474519 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031a3fd8-e28c-4476-abba-c26d852a7477 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:25.475174 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031a3fd8-e28c-4476-abba-c26d852a7477 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:25.475452 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031a3fd8-e28c-4476-abba-c26d852a7477 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:25.475830 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031a3fd8-e28c-4476-abba-c26d852a7477 None admin] PATH_INFO: `/v3/users/549341ef566b4758821e81a97b2f848e` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:25.477005 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031a3fd8-e28c-4476-abba-c26d852a7477 None admin] RBAC: Authorizing `identity:get_user(user_id=549341ef566b4758821e81a97b2f848e)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:25.478566 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031a3fd8-e28c-4476-abba-c26d852a7477 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:25.480200 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 127/254] 10.222.0.42 () {56 vars in 1177 bytes} [Thu Oct 27 01:11:25 2022] GET /identity/v3/users/549341ef566b4758821e81a97b2f848e => generated 293 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:25.486312 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc53b386-cedf-4d66-b459-8a1bb70b25b2 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:25.590804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:25.601100 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc53b386-cedf-4d66-b459-8a1bb70b25b2 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:25.638037 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda55e70-2df9-4bd2-8c8d-f838bebaf565 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:25.638660 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda55e70-2df9-4bd2-8c8d-f838bebaf565 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:25.638939 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda55e70-2df9-4bd2-8c8d-f838bebaf565 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:25.639261 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda55e70-2df9-4bd2-8c8d-f838bebaf565 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:25.640002 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda55e70-2df9-4bd2-8c8d-f838bebaf565 None admin] RBAC: Adding query filter params (role.id=f199b79a7178462fa67bb8243943408f, user.id=549341ef566b4758821e81a97b2f848e, scope.system=all) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:25.640364 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda55e70-2df9-4bd2-8c8d-f838bebaf565 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:25.641872 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda55e70-2df9-4bd2-8c8d-f838bebaf565 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:25.655457 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 128/255] 10.222.0.42 () {56 vars in 1308 bytes} [Thu Oct 27 01:11:25 2022] GET /identity/v3/role_assignments?user.id=549341ef566b4758821e81a97b2f848e&scope.system=all&role.id=f199b79a7178462fa67bb8243943408f => generated 510 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:27.465710 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd41124-056c-411a-be3a-eb6d615d909b None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:27.466306 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd41124-056c-411a-be3a-eb6d615d909b None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:27.466306 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd41124-056c-411a-be3a-eb6d615d909b None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:27.467851 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 128/256] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:27.482954 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65823a3-c43e-46bf-b5b6-19da6e5158f6 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:27.483431 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65823a3-c43e-46bf-b5b6-19da6e5158f6 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:27.483532 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65823a3-c43e-46bf-b5b6-19da6e5158f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:27.510016 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-d65823a3-c43e-46bf-b5b6-19da6e5158f6 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:27.511233 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d65823a3-c43e-46bf-b5b6-19da6e5158f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:27.592948 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 129/257] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:27 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 27 01:11:27.705869 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3515460-bc9b-4258-a657-64b13d039c6a None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:27.706210 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3515460-bc9b-4258-a657-64b13d039c6a None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:27.706422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3515460-bc9b-4258-a657-64b13d039c6a None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:27.708602 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 129/258] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:27.715010 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa308a8-d6ae-472a-9ae1-374dd930094e None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:27.715304 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa308a8-d6ae-472a-9ae1-374dd930094e None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:27.715582 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa308a8-d6ae-472a-9ae1-374dd930094e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:27.742104 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-7aa308a8-d6ae-472a-9ae1-374dd930094e None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:27.743248 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-7aa308a8-d6ae-472a-9ae1-374dd930094e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:27.751499 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 130/259] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:27 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 27 01:11:28.151811 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3515460-bc9b-4258-a657-64b13d039c6a None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:28.169173 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3515460-bc9b-4258-a657-64b13d039c6a None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:28.173977 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f44fd43-55c7-4f42-8a7d-bd6b7b47bc3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:28.174856 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f44fd43-55c7-4f42-8a7d-bd6b7b47bc3b admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:28.175328 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f44fd43-55c7-4f42-8a7d-bd6b7b47bc3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:28.175811 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f44fd43-55c7-4f42-8a7d-bd6b7b47bc3b admin admin] PATH_INFO: `/v3/roles/member` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:28.195292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f44fd43-55c7-4f42-8a7d-bd6b7b47bc3b admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:28.200527 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-9f44fd43-55c7-4f42-8a7d-bd6b7b47bc3b admin admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 27 01:11:28.200527 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:28.200527 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:28.200527 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:28.200527 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:28.200527 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:28.200527 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:28.200527 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:28.200527 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:28.200527 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:28.200527 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:28.200527 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:28.200527 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:28.200527 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:28.200527 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:28.200527 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:28.201415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:28.201415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:28.201415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:28.201415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:28.201415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:28.201415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:28.201415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:28.201415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:28.201415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:28.201415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:28.201415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:28.201415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:28.201415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:28.201415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:28.201415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:28.201415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:28.202212 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:28.202212 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:28.202212 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:28.202212 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:28.202212 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:28.202212 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:28.202212 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:28.202212 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:28.202212 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:28.202212 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:28.202212 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 27 01:11:28.202212 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:28.204457 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 130/260] 10.222.0.42 () {56 vars in 1120 bytes} [Thu Oct 27 01:11:28 2022] GET /identity/v3/roles/member => generated 84 bytes in 53 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:28.211648 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa308a8-d6ae-472a-9ae1-374dd930094e None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:28.221513 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa308a8-d6ae-472a-9ae1-374dd930094e None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:28.229019 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872fcdb0-d040-44c8-8851-c6408ae9b204 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:28.229911 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872fcdb0-d040-44c8-8851-c6408ae9b204 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:28.230385 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872fcdb0-d040-44c8-8851-c6408ae9b204 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:28.230875 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872fcdb0-d040-44c8-8851-c6408ae9b204 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:28.231752 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-872fcdb0-d040-44c8-8851-c6408ae9b204 admin admin] RBAC: Adding query filter params (name=member) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:28.232246 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-872fcdb0-d040-44c8-8851-c6408ae9b204 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:28.234269 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: WARNING py.warnings [None req-872fcdb0-d040-44c8-8851-c6408ae9b204 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 27 01:11:28.234269 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: warnings.warn(msg) Oct 27 01:11:28.234269 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: Oct 27 01:11:28.235619 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-872fcdb0-d040-44c8-8851-c6408ae9b204 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:28.247639 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 131/261] 10.222.0.42 () {56 vars in 1122 bytes} [Thu Oct 27 01:11:28 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:28.253496 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f44fd43-55c7-4f42-8a7d-bd6b7b47bc3b admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:28.264174 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f44fd43-55c7-4f42-8a7d-bd6b7b47bc3b admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:28.272419 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de595ffb-6478-49f8-a29c-f9931fc5dd8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:28.273391 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de595ffb-6478-49f8-a29c-f9931fc5dd8e admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:28.273881 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de595ffb-6478-49f8-a29c-f9931fc5dd8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:28.274349 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de595ffb-6478-49f8-a29c-f9931fc5dd8e admin admin] PATH_INFO: `/v3/groups/3215d70f63f84c98a348daded668804e` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:28.281499 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de595ffb-6478-49f8-a29c-f9931fc5dd8e admin admin] RBAC: Authorizing `identity:get_group(group_id=3215d70f63f84c98a348daded668804e)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:28.283552 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: WARNING py.warnings [None req-de595ffb-6478-49f8-a29c-f9931fc5dd8e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:get_group": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.group.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 27 01:11:28.283552 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: warnings.warn(msg) Oct 27 01:11:28.283552 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: Oct 27 01:11:28.285177 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de595ffb-6478-49f8-a29c-f9931fc5dd8e admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:28.287163 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 131/262] 10.222.0.42 () {56 vars in 1201 bytes} [Thu Oct 27 01:11:28 2022] GET /identity/v3/groups/3215d70f63f84c98a348daded668804e => generated 226 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:28.292705 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872fcdb0-d040-44c8-8851-c6408ae9b204 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:28.303863 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872fcdb0-d040-44c8-8851-c6408ae9b204 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:28.313123 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ad76d0-ef2e-4a24-99e0-4bcb82889e5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:28.314014 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ad76d0-ef2e-4a24-99e0-4bcb82889e5e admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:28.314506 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ad76d0-ef2e-4a24-99e0-4bcb82889e5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:28.314974 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ad76d0-ef2e-4a24-99e0-4bcb82889e5e admin admin] PATH_INFO: `/v3/projects/50a39f634ddf41798003dd395f14be68` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:28.316322 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3ad76d0-ef2e-4a24-99e0-4bcb82889e5e admin admin] RBAC: Authorizing `identity:get_project(project_id=50a39f634ddf41798003dd395f14be68)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:28.318846 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3ad76d0-ef2e-4a24-99e0-4bcb82889e5e admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:28.320755 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 132/263] 10.222.0.42 () {56 vars in 1207 bytes} [Thu Oct 27 01:11:28 2022] GET /identity/v3/projects/50a39f634ddf41798003dd395f14be68 => generated 298 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:28.326531 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de595ffb-6478-49f8-a29c-f9931fc5dd8e admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:28.334289 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ad76d0-ef2e-4a24-99e0-4bcb82889e5e admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:28.339025 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de595ffb-6478-49f8-a29c-f9931fc5dd8e admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:28.355852 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1fe24b-bba0-44c5-815d-e8db90214321 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:28.356917 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1fe24b-bba0-44c5-815d-e8db90214321 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:28.357422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1fe24b-bba0-44c5-815d-e8db90214321 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:28.357892 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1fe24b-bba0-44c5-815d-e8db90214321 admin admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:28.368234 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ad76d0-ef2e-4a24-99e0-4bcb82889e5e admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:28.377493 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f632cb98-bd45-4470-b427-92600441cc4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:28.380520 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f632cb98-bd45-4470-b427-92600441cc4b admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:28.382240 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f632cb98-bd45-4470-b427-92600441cc4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:28.385050 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f632cb98-bd45-4470-b427-92600441cc4b admin admin] PATH_INFO: `/v3/projects/50a39f634ddf41798003dd395f14be68/groups/3215d70f63f84c98a348daded668804e/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:28.387691 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f632cb98-bd45-4470-b427-92600441cc4b admin admin] RBAC: Authorizing `identity:create_grant(project_id=50a39f634ddf41798003dd395f14be68, group_id=3215d70f63f84c98a348daded668804e, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:28.391695 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b1fe24b-bba0-44c5-815d-e8db90214321 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:28.392531 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: WARNING py.warnings [None req-f632cb98-bd45-4470-b427-92600441cc4b admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 27 01:11:28.392531 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: warnings.warn(msg) Oct 27 01:11:28.392531 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: Oct 27 01:11:28.398700 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-4b1fe24b-bba0-44c5-815d-e8db90214321 admin admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 27 01:11:28.398700 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:28.398700 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:28.398700 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:28.398700 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:28.398700 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:28.398700 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:28.398700 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:28.398700 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:28.398700 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:28.398700 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:28.398700 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:28.398700 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:28.398700 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:28.398700 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:28.398700 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:28.399649 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:28.399649 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:28.399649 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:28.399649 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:28.399649 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:28.399649 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:28.399649 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:28.399649 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:28.399649 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:28.399649 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:28.399649 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:28.399649 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:28.399649 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:28.399649 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:28.399649 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:28.399649 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:28.400430 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:28.400430 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:28.400430 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:28.400430 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:28.400430 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:28.400430 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:28.400430 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:28.400430 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:28.400430 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:28.400430 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:28.400430 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 27 01:11:28.400430 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:28.403952 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f632cb98-bd45-4470-b427-92600441cc4b admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:28.413706 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 132/264] 10.222.0.42 () {56 vars in 1135 bytes} [Thu Oct 27 01:11:28 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 87 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:28.419653 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1fe24b-bba0-44c5-815d-e8db90214321 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:28.433983 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 133/265] 10.222.0.42 () {58 vars in 1463 bytes} [Thu Oct 27 01:11:28 2022] PUT /identity/v3/projects/50a39f634ddf41798003dd395f14be68/groups/3215d70f63f84c98a348daded668804e/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:11:28.441579 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1fe24b-bba0-44c5-815d-e8db90214321 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:28.500242 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:28.514419 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-229de1a3-7465-4bc0-96af-b97757fe6d73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:28.515064 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229de1a3-7465-4bc0-96af-b97757fe6d73 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:28.515344 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229de1a3-7465-4bc0-96af-b97757fe6d73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:28.515617 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229de1a3-7465-4bc0-96af-b97757fe6d73 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:28.516327 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-229de1a3-7465-4bc0-96af-b97757fe6d73 admin admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:28.516635 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-229de1a3-7465-4bc0-96af-b97757fe6d73 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:28.517799 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: WARNING py.warnings [None req-229de1a3-7465-4bc0-96af-b97757fe6d73 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 27 01:11:28.517799 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: warnings.warn(msg) Oct 27 01:11:28.517799 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: Oct 27 01:11:28.518783 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-229de1a3-7465-4bc0-96af-b97757fe6d73 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:28.531412 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 133/266] 10.222.0.42 () {56 vars in 1132 bytes} [Thu Oct 27 01:11:28 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 112 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:11:28.537745 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f632cb98-bd45-4470-b427-92600441cc4b admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:28.545728 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b5412e-ea61-4e25-a886-a9cca5a684e4 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:28.546043 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b5412e-ea61-4e25-a886-a9cca5a684e4 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:28.546274 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b5412e-ea61-4e25-a886-a9cca5a684e4 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:28.547813 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 134/267] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:28.552788 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f632cb98-bd45-4470-b427-92600441cc4b admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:28.555906 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393c850e-5c51-499f-9460-3946dc1f86ba None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:28.556206 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393c850e-5c51-499f-9460-3946dc1f86ba None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:28.556702 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393c850e-5c51-499f-9460-3946dc1f86ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:28.563300 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5458ad23-130a-4474-8c97-eafb0b62500b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:28.564003 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5458ad23-130a-4474-8c97-eafb0b62500b admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:28.564295 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5458ad23-130a-4474-8c97-eafb0b62500b admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:28.564563 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5458ad23-130a-4474-8c97-eafb0b62500b admin admin] PATH_INFO: `/v3/groups/3215d70f63f84c98a348daded668804e` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:28.565823 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5458ad23-130a-4474-8c97-eafb0b62500b admin admin] RBAC: Authorizing `identity:get_group(group_id=3215d70f63f84c98a348daded668804e)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:28.567599 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: WARNING py.warnings [None req-5458ad23-130a-4474-8c97-eafb0b62500b admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:get_group": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.group.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 27 01:11:28.567599 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: warnings.warn(msg) Oct 27 01:11:28.567599 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: Oct 27 01:11:28.568831 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5458ad23-130a-4474-8c97-eafb0b62500b admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:28.570829 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 134/268] 10.222.0.42 () {56 vars in 1201 bytes} [Thu Oct 27 01:11:28 2022] GET /identity/v3/groups/3215d70f63f84c98a348daded668804e => generated 226 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:11:28.577797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5458ad23-130a-4474-8c97-eafb0b62500b admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:28.590244 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-393c850e-5c51-499f-9460-3946dc1f86ba None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:28.591461 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-393c850e-5c51-499f-9460-3946dc1f86ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:28.594935 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5458ad23-130a-4474-8c97-eafb0b62500b admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:28.600723 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a45eae7b-a3e6-4cac-923e-a776dcaa357c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:28.601474 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45eae7b-a3e6-4cac-923e-a776dcaa357c admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:28.601828 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45eae7b-a3e6-4cac-923e-a776dcaa357c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:28.602153 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45eae7b-a3e6-4cac-923e-a776dcaa357c admin admin] PATH_INFO: `/v3/projects/50a39f634ddf41798003dd395f14be68` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:28.603437 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a45eae7b-a3e6-4cac-923e-a776dcaa357c admin admin] RBAC: Authorizing `identity:get_project(project_id=50a39f634ddf41798003dd395f14be68)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:28.605458 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 135/269] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:28 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 27 01:11:28.606334 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a45eae7b-a3e6-4cac-923e-a776dcaa357c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:28.607177 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0783468f-3046-439f-98e9-be05aab6174c None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:28.607690 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0783468f-3046-439f-98e9-be05aab6174c None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:28.608151 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0783468f-3046-439f-98e9-be05aab6174c None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:28.608251 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 135/270] 10.222.0.42 () {56 vars in 1207 bytes} [Thu Oct 27 01:11:28 2022] GET /identity/v3/projects/50a39f634ddf41798003dd395f14be68 => generated 298 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:28.609791 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 136/271] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:28.614239 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a45eae7b-a3e6-4cac-923e-a776dcaa357c admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:28.615668 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc009811-eeb5-4e2a-b7a0-4efb514fe23b None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:28.616181 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc009811-eeb5-4e2a-b7a0-4efb514fe23b None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:28.616680 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc009811-eeb5-4e2a-b7a0-4efb514fe23b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:28.630943 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a45eae7b-a3e6-4cac-923e-a776dcaa357c admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:28.642234 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4903976-19b6-41e2-a642-c19243980c12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:28.643054 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4903976-19b6-41e2-a642-c19243980c12 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:28.643342 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4903976-19b6-41e2-a642-c19243980c12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:28.643618 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4903976-19b6-41e2-a642-c19243980c12 admin admin] PATH_INFO: `/v3/projects/50a39f634ddf41798003dd395f14be68/groups/3215d70f63f84c98a348daded668804e/roles/bb9bd8ef24ff4aa5b639a2bd8567345b` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:28.649482 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4903976-19b6-41e2-a642-c19243980c12 admin admin] RBAC: Authorizing `identity:create_grant(project_id=50a39f634ddf41798003dd395f14be68, group_id=3215d70f63f84c98a348daded668804e, role_id=bb9bd8ef24ff4aa5b639a2bd8567345b)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:28.656750 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4903976-19b6-41e2-a642-c19243980c12 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:28.660202 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-cc009811-eeb5-4e2a-b7a0-4efb514fe23b None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:28.663514 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-cc009811-eeb5-4e2a-b7a0-4efb514fe23b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:28.684355 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 136/272] 10.222.0.42 () {58 vars in 1463 bytes} [Thu Oct 27 01:11:28 2022] PUT /identity/v3/projects/50a39f634ddf41798003dd395f14be68/groups/3215d70f63f84c98a348daded668804e/roles/bb9bd8ef24ff4aa5b639a2bd8567345b => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:11:28.720630 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d089464-942c-4442-8bdc-0132df388a86 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:28.720937 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d089464-942c-4442-8bdc-0132df388a86 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:28.722751 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d089464-942c-4442-8bdc-0132df388a86 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:28.724926 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 137/273] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:28 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 27 01:11:28.746628 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddd0749-7c9e-4175-9f44-bb9602926030 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:28.747005 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddd0749-7c9e-4175-9f44-bb9602926030 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:28.747169 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddd0749-7c9e-4175-9f44-bb9602926030 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:28.776000 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 137/274] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 27 01:11:28.785579 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-fddd0749-7c9e-4175-9f44-bb9602926030 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:28.787878 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-fddd0749-7c9e-4175-9f44-bb9602926030 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:28.855477 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:28.880073 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 138/275] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:28 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 27 01:11:29.045535 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc009811-eeb5-4e2a-b7a0-4efb514fe23b None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:29.058782 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc009811-eeb5-4e2a-b7a0-4efb514fe23b None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:29.063099 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b6a5ad-afff-4cad-a7f8-3a0dcc48de1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:29.063733 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b6a5ad-afff-4cad-a7f8-3a0dcc48de1d admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:29.064003 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b6a5ad-afff-4cad-a7f8-3a0dcc48de1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:29.064275 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b6a5ad-afff-4cad-a7f8-3a0dcc48de1d admin admin] PATH_INFO: `/v3/roles/member` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:29.080999 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b6a5ad-afff-4cad-a7f8-3a0dcc48de1d admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:29.086596 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-c5b6a5ad-afff-4cad-a7f8-3a0dcc48de1d admin admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 27 01:11:29.086596 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:29.086596 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:29.086596 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:29.086596 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:29.086596 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:29.086596 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:29.086596 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:29.086596 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:29.086596 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:29.086596 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:29.086596 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:29.086596 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:29.086596 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:29.086596 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:29.086596 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:29.087491 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:29.087491 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:29.087491 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:29.087491 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:29.087491 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:29.087491 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:29.087491 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:29.087491 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:29.087491 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:29.087491 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:29.087491 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:29.087491 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:29.087491 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:29.087491 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:29.087491 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:29.087491 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:29.088297 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:29.088297 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:29.088297 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:29.088297 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:29.088297 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:29.088297 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:29.088297 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:29.088297 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:29.088297 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:29.088297 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:29.088297 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 27 01:11:29.088297 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:29.088297 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 138/276] 10.222.0.42 () {56 vars in 1120 bytes} [Thu Oct 27 01:11:29 2022] GET /identity/v3/roles/member => generated 84 bytes in 43 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:29.093606 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddd0749-7c9e-4175-9f44-bb9602926030 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:29.106303 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddd0749-7c9e-4175-9f44-bb9602926030 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:29.114233 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e274967d-f191-40da-bda0-b47e1af46047 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:29.114916 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e274967d-f191-40da-bda0-b47e1af46047 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:29.115201 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e274967d-f191-40da-bda0-b47e1af46047 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:29.115467 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e274967d-f191-40da-bda0-b47e1af46047 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:29.116147 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e274967d-f191-40da-bda0-b47e1af46047 admin admin] RBAC: Adding query filter params (name=member) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:29.116452 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e274967d-f191-40da-bda0-b47e1af46047 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:29.118079 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e274967d-f191-40da-bda0-b47e1af46047 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:29.127217 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 139/277] 10.222.0.42 () {56 vars in 1122 bytes} [Thu Oct 27 01:11:29 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:29.133453 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b6a5ad-afff-4cad-a7f8-3a0dcc48de1d admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:29.143904 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b6a5ad-afff-4cad-a7f8-3a0dcc48de1d admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:29.153082 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff0dc959-8ef3-4ac4-beae-692904d501eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:29.153719 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0dc959-8ef3-4ac4-beae-692904d501eb admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:29.153993 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0dc959-8ef3-4ac4-beae-692904d501eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:29.154256 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0dc959-8ef3-4ac4-beae-692904d501eb admin admin] PATH_INFO: `/v3/groups/3215d70f63f84c98a348daded668804e` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:29.155372 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff0dc959-8ef3-4ac4-beae-692904d501eb admin admin] RBAC: Authorizing `identity:get_group(group_id=3215d70f63f84c98a348daded668804e)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:29.157314 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff0dc959-8ef3-4ac4-beae-692904d501eb admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:29.158871 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 139/278] 10.222.0.42 () {56 vars in 1201 bytes} [Thu Oct 27 01:11:29 2022] GET /identity/v3/groups/3215d70f63f84c98a348daded668804e => generated 226 bytes in 26 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:29.165727 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e274967d-f191-40da-bda0-b47e1af46047 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:29.174907 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff0dc959-8ef3-4ac4-beae-692904d501eb admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:29.178822 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e274967d-f191-40da-bda0-b47e1af46047 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:29.186590 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-004998f8-7cfd-4aca-bbd5-994e1cd8ec2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:29.187227 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004998f8-7cfd-4aca-bbd5-994e1cd8ec2b admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:29.187494 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004998f8-7cfd-4aca-bbd5-994e1cd8ec2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:29.187767 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004998f8-7cfd-4aca-bbd5-994e1cd8ec2b admin admin] PATH_INFO: `/v3/projects/69bfdc060ff949e2ba795cad23f9f746` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:29.188909 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-004998f8-7cfd-4aca-bbd5-994e1cd8ec2b admin admin] RBAC: Authorizing `identity:get_project(project_id=69bfdc060ff949e2ba795cad23f9f746)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:29.191019 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-004998f8-7cfd-4aca-bbd5-994e1cd8ec2b admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:29.192811 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 140/279] 10.222.0.42 () {56 vars in 1207 bytes} [Thu Oct 27 01:11:29 2022] GET /identity/v3/projects/69bfdc060ff949e2ba795cad23f9f746 => generated 302 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:29.203076 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-004998f8-7cfd-4aca-bbd5-994e1cd8ec2b admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:29.210449 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-004998f8-7cfd-4aca-bbd5-994e1cd8ec2b admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:29.221139 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096d2007-dd69-4e97-9b68-8956e9584760 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:29.221139 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096d2007-dd69-4e97-9b68-8956e9584760 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:29.221139 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096d2007-dd69-4e97-9b68-8956e9584760 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:29.221139 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096d2007-dd69-4e97-9b68-8956e9584760 admin admin] PATH_INFO: `/v3/projects/69bfdc060ff949e2ba795cad23f9f746/groups/3215d70f63f84c98a348daded668804e/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:29.221486 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096d2007-dd69-4e97-9b68-8956e9584760 admin admin] RBAC: Authorizing `identity:create_grant(project_id=69bfdc060ff949e2ba795cad23f9f746, group_id=3215d70f63f84c98a348daded668804e, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:29.221732 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096d2007-dd69-4e97-9b68-8956e9584760 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:29.249148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 141/280] 10.222.0.42 () {58 vars in 1463 bytes} [Thu Oct 27 01:11:29 2022] PUT /identity/v3/projects/69bfdc060ff949e2ba795cad23f9f746/groups/3215d70f63f84c98a348daded668804e/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:11:29.321739 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:29.328216 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff0dc959-8ef3-4ac4-beae-692904d501eb admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:29.351116 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096d2007-dd69-4e97-9b68-8956e9584760 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:29.430540 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33baf726-7c05-43fc-9333-98766b5ce53b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:29.431198 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33baf726-7c05-43fc-9333-98766b5ce53b None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:29.431723 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33baf726-7c05-43fc-9333-98766b5ce53b None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:29.431994 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33baf726-7c05-43fc-9333-98766b5ce53b None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:29.446105 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33baf726-7c05-43fc-9333-98766b5ce53b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:29.452156 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-33baf726-7c05-43fc-9333-98766b5ce53b None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 27 01:11:29.452156 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:29.452156 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:29.452156 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:29.452156 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:29.452156 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:29.452156 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:29.452156 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:29.452156 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:29.452156 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:29.452156 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:29.452156 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:29.452156 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:29.452156 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:29.452156 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:29.452156 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:29.453005 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:29.453005 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:29.453005 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:29.453005 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:29.453005 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:29.453005 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:29.453005 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:29.453005 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:29.453005 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:29.453005 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:29.453005 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:29.453005 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:29.453005 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:29.453005 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:29.453005 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:29.453005 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:29.453860 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:29.453860 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:29.453860 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:29.453860 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:29.453860 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:29.453860 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:29.453860 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:29.453860 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:29.453860 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:29.453860 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:29.453860 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 27 01:11:29.453860 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:29.456535 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 140/281] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:11:29 2022] GET /identity/v3/roles/reader => generated 84 bytes in 283 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:29.456723 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096d2007-dd69-4e97-9b68-8956e9584760 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:29.464506 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33baf726-7c05-43fc-9333-98766b5ce53b None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:29.469393 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5c07fb-6520-452a-8508-a93668f9aaa5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:29.471312 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5c07fb-6520-452a-8508-a93668f9aaa5 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:29.471630 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5c07fb-6520-452a-8508-a93668f9aaa5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:29.471914 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5c07fb-6520-452a-8508-a93668f9aaa5 admin admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:29.486566 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba5c07fb-6520-452a-8508-a93668f9aaa5 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:29.494114 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-ba5c07fb-6520-452a-8508-a93668f9aaa5 admin admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 27 01:11:29.494114 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:29.494114 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:29.494114 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:29.494114 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:29.494114 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:29.494114 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:29.494114 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:29.494114 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:29.494114 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:29.494114 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:29.494114 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:29.494114 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:29.494114 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:29.494114 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:29.494114 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:29.495053 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:29.495053 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:29.495053 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:29.495053 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:29.495053 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:29.495053 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:29.495053 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:29.495053 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:29.495053 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:29.495053 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:29.495053 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:29.495053 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:29.495053 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:29.495053 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:29.495053 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:29.495053 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:29.495960 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:29.495960 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:29.495960 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:29.495960 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:29.495960 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:29.495960 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:29.495960 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:29.495960 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:29.495960 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:29.495960 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:29.495960 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 27 01:11:29.495960 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:29.495960 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 142/282] 10.222.0.42 () {56 vars in 1135 bytes} [Thu Oct 27 01:11:29 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 144 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 27 01:11:29.501125 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5c07fb-6520-452a-8508-a93668f9aaa5 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:29.521663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5c07fb-6520-452a-8508-a93668f9aaa5 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:29.536778 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40aa6386-87a6-49b9-a69d-3b81da79a6df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:29.537962 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40aa6386-87a6-49b9-a69d-3b81da79a6df admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:29.538852 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40aa6386-87a6-49b9-a69d-3b81da79a6df admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:29.539313 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40aa6386-87a6-49b9-a69d-3b81da79a6df admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:29.540263 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40aa6386-87a6-49b9-a69d-3b81da79a6df admin admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:29.540777 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40aa6386-87a6-49b9-a69d-3b81da79a6df admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:29.544623 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40aa6386-87a6-49b9-a69d-3b81da79a6df admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:29.564681 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 143/283] 10.222.0.42 () {56 vars in 1132 bytes} [Thu Oct 27 01:11:29 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:11:29.568616 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40aa6386-87a6-49b9-a69d-3b81da79a6df admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:29.586603 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40aa6386-87a6-49b9-a69d-3b81da79a6df admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:29.600414 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe9cfe3-eb79-45cb-a033-e5bf7d57a8d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:29.601089 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe9cfe3-eb79-45cb-a033-e5bf7d57a8d0 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:29.601368 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe9cfe3-eb79-45cb-a033-e5bf7d57a8d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:29.601637 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe9cfe3-eb79-45cb-a033-e5bf7d57a8d0 admin admin] PATH_INFO: `/v3/groups/3215d70f63f84c98a348daded668804e` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:29.603772 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe9cfe3-eb79-45cb-a033-e5bf7d57a8d0 admin admin] RBAC: Authorizing `identity:get_group(group_id=3215d70f63f84c98a348daded668804e)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:29.605896 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe9cfe3-eb79-45cb-a033-e5bf7d57a8d0 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:29.607947 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 144/284] 10.222.0.42 () {56 vars in 1201 bytes} [Thu Oct 27 01:11:29 2022] GET /identity/v3/groups/3215d70f63f84c98a348daded668804e => generated 226 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:29.613530 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe9cfe3-eb79-45cb-a033-e5bf7d57a8d0 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:29.633117 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe9cfe3-eb79-45cb-a033-e5bf7d57a8d0 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:29.646593 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33baf726-7c05-43fc-9333-98766b5ce53b None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:29.653738 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9374a38-a0ef-4d85-b0d5-7ace874232c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:29.654383 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9374a38-a0ef-4d85-b0d5-7ace874232c0 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:29.654662 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9374a38-a0ef-4d85-b0d5-7ace874232c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:29.654934 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9374a38-a0ef-4d85-b0d5-7ace874232c0 admin admin] PATH_INFO: `/v3/projects/69bfdc060ff949e2ba795cad23f9f746` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:29.656121 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9374a38-a0ef-4d85-b0d5-7ace874232c0 admin admin] RBAC: Authorizing `identity:get_project(project_id=69bfdc060ff949e2ba795cad23f9f746)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:29.658226 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9374a38-a0ef-4d85-b0d5-7ace874232c0 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:29.660212 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 145/285] 10.222.0.42 () {56 vars in 1207 bytes} [Thu Oct 27 01:11:29 2022] GET /identity/v3/projects/69bfdc060ff949e2ba795cad23f9f746 => generated 302 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:11:29.666248 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9374a38-a0ef-4d85-b0d5-7ace874232c0 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:29.691214 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9374a38-a0ef-4d85-b0d5-7ace874232c0 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:29.691641 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4506ec-cf96-4ac8-bc3e-a0bf5940d88f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:29.692849 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4506ec-cf96-4ac8-bc3e-a0bf5940d88f None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:29.693376 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4506ec-cf96-4ac8-bc3e-a0bf5940d88f None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:29.693837 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4506ec-cf96-4ac8-bc3e-a0bf5940d88f None admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:29.694792 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be4506ec-cf96-4ac8-bc3e-a0bf5940d88f None admin] RBAC: Adding query filter params (name=reader) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:29.695298 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be4506ec-cf96-4ac8-bc3e-a0bf5940d88f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:29.696951 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be4506ec-cf96-4ac8-bc3e-a0bf5940d88f None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:29.711163 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195e5289-3a49-40d9-b826-21006f451446 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:29.711163 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195e5289-3a49-40d9-b826-21006f451446 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:29.711163 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195e5289-3a49-40d9-b826-21006f451446 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:29.711163 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195e5289-3a49-40d9-b826-21006f451446 admin admin] PATH_INFO: `/v3/projects/69bfdc060ff949e2ba795cad23f9f746/groups/3215d70f63f84c98a348daded668804e/roles/bb9bd8ef24ff4aa5b639a2bd8567345b` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:29.712800 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 141/286] 10.222.0.42 () {56 vars in 1101 bytes} [Thu Oct 27 01:11:29 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:29.714257 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-195e5289-3a49-40d9-b826-21006f451446 admin admin] RBAC: Authorizing `identity:create_grant(project_id=69bfdc060ff949e2ba795cad23f9f746, group_id=3215d70f63f84c98a348daded668804e, role_id=bb9bd8ef24ff4aa5b639a2bd8567345b)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:29.716937 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-195e5289-3a49-40d9-b826-21006f451446 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:29.718383 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4506ec-cf96-4ac8-bc3e-a0bf5940d88f None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:29.743062 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:29.763253 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 146/287] 10.222.0.42 () {58 vars in 1463 bytes} [Thu Oct 27 01:11:29 2022] PUT /identity/v3/projects/69bfdc060ff949e2ba795cad23f9f746/groups/3215d70f63f84c98a348daded668804e/roles/bb9bd8ef24ff4aa5b639a2bd8567345b => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:11:29.863556 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4506ec-cf96-4ac8-bc3e-a0bf5940d88f None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:29.905542 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe41912-7ad6-4bfd-9474-f2942d3649d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:29.906424 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe41912-7ad6-4bfd-9474-f2942d3649d3 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:29.906867 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe41912-7ad6-4bfd-9474-f2942d3649d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:29.907278 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe41912-7ad6-4bfd-9474-f2942d3649d3 None admin] PATH_INFO: `/v3/users/6cd9a9702f5b4db2b10e8e0de4f20bad` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:29.908599 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe41912-7ad6-4bfd-9474-f2942d3649d3 None admin] RBAC: Authorizing `identity:get_user(user_id=6cd9a9702f5b4db2b10e8e0de4f20bad)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:29.910256 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe41912-7ad6-4bfd-9474-f2942d3649d3 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:29.912452 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 142/288] 10.222.0.42 () {56 vars in 1177 bytes} [Thu Oct 27 01:11:29 2022] GET /identity/v3/users/6cd9a9702f5b4db2b10e8e0de4f20bad => generated 293 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:11:29.920047 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195e5289-3a49-40d9-b826-21006f451446 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:30.031206 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:30.072309 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195e5289-3a49-40d9-b826-21006f451446 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:30.120548 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b47a01-3bca-42f9-963c-4134dee81e42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:30.121124 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b47a01-3bca-42f9-963c-4134dee81e42 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:30.121410 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b47a01-3bca-42f9-963c-4134dee81e42 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:30.121686 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b47a01-3bca-42f9-963c-4134dee81e42 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:30.122437 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b47a01-3bca-42f9-963c-4134dee81e42 None admin] RBAC: Adding query filter params (role.id=5fa0c579f1fb428e9dcae929dd21846f, user.id=6cd9a9702f5b4db2b10e8e0de4f20bad, scope.system=all) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:30.122747 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b47a01-3bca-42f9-963c-4134dee81e42 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:30.124210 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b47a01-3bca-42f9-963c-4134dee81e42 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:30.141254 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 147/289] 10.222.0.42 () {56 vars in 1308 bytes} [Thu Oct 27 01:11:29 2022] GET /identity/v3/role_assignments?user.id=6cd9a9702f5b4db2b10e8e0de4f20bad&scope.system=all&role.id=5fa0c579f1fb428e9dcae929dd21846f => generated 510 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:30.511633 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f744b2d3-b163-4c44-aa3f-060300f7849f None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:30.512277 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f744b2d3-b163-4c44-aa3f-060300f7849f None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:30.512753 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f744b2d3-b163-4c44-aa3f-060300f7849f None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:30.514305 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 143/290] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:30.520049 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a28eab4-c33f-4553-b803-d3d93f532ec6 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:30.520381 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a28eab4-c33f-4553-b803-d3d93f532ec6 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:30.520608 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a28eab4-c33f-4553-b803-d3d93f532ec6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:30.550041 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-6a28eab4-c33f-4553-b803-d3d93f532ec6 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:30.551955 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-6a28eab4-c33f-4553-b803-d3d93f532ec6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:30.620911 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 148/291] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:30 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Oct 27 01:11:31.060361 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f744b2d3-b163-4c44-aa3f-060300f7849f None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:31.081878 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f744b2d3-b163-4c44-aa3f-060300f7849f None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:31.086963 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e54d74b-881f-44b2-a218-1214724ac651 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:31.087852 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e54d74b-881f-44b2-a218-1214724ac651 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:31.088326 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e54d74b-881f-44b2-a218-1214724ac651 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:31.088786 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e54d74b-881f-44b2-a218-1214724ac651 admin admin] PATH_INFO: `/v3/roles/admin` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:31.104260 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e54d74b-881f-44b2-a218-1214724ac651 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:31.109397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-0e54d74b-881f-44b2-a218-1214724ac651 admin admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:11:31.109397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:31.109397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:31.109397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:31.109397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:31.109397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:31.109397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:31.109397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:31.109397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:31.109397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:31.109397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:31.109397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:31.109397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:31.109397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:31.109397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:31.109397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:31.110258 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:31.110258 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:31.110258 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:31.110258 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:31.110258 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:31.110258 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:31.110258 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:31.110258 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:31.110258 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:31.110258 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:31.110258 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:31.110258 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:31.110258 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:31.110258 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:31.110258 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:31.110258 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:31.111036 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:31.111036 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:31.111036 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:31.111036 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:31.111036 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:31.111036 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:31.111036 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:31.111036 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:31.111036 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:31.111036 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:31.111036 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:11:31.111036 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:31.117797 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 144/292] 10.222.0.42 () {56 vars in 1117 bytes} [Thu Oct 27 01:11:31 2022] GET /identity/v3/roles/admin => generated 83 bytes in 58 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:31.123694 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a28eab4-c33f-4553-b803-d3d93f532ec6 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:31.133763 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a28eab4-c33f-4553-b803-d3d93f532ec6 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:31.140052 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d60670-399a-45b7-8646-e0d29c4773b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:31.140944 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d60670-399a-45b7-8646-e0d29c4773b1 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:31.141478 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d60670-399a-45b7-8646-e0d29c4773b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:31.141940 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d60670-399a-45b7-8646-e0d29c4773b1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:31.142915 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23d60670-399a-45b7-8646-e0d29c4773b1 admin admin] RBAC: Adding query filter params (name=admin) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:31.143407 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23d60670-399a-45b7-8646-e0d29c4773b1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:31.145961 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23d60670-399a-45b7-8646-e0d29c4773b1 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:31.155451 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 149/293] 10.222.0.42 () {56 vars in 1120 bytes} [Thu Oct 27 01:11:31 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:31.161470 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e54d74b-881f-44b2-a218-1214724ac651 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:31.172602 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e54d74b-881f-44b2-a218-1214724ac651 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:31.180703 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbedd35f-fce9-4061-8d36-623bb30f909b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:31.181984 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbedd35f-fce9-4061-8d36-623bb30f909b admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:31.183892 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbedd35f-fce9-4061-8d36-623bb30f909b admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:31.184359 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbedd35f-fce9-4061-8d36-623bb30f909b admin admin] PATH_INFO: `/v3/groups/1bdbf8d0f4684cf59bcd5fb9b6ada058` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:31.190240 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbedd35f-fce9-4061-8d36-623bb30f909b admin admin] RBAC: Authorizing `identity:get_group(group_id=1bdbf8d0f4684cf59bcd5fb9b6ada058)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:31.192582 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbedd35f-fce9-4061-8d36-623bb30f909b admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:31.194866 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 145/294] 10.222.0.42 () {56 vars in 1201 bytes} [Thu Oct 27 01:11:31 2022] GET /identity/v3/groups/1bdbf8d0f4684cf59bcd5fb9b6ada058 => generated 229 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:31.200095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d60670-399a-45b7-8646-e0d29c4773b1 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:31.211690 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d60670-399a-45b7-8646-e0d29c4773b1 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:31.216688 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7ab41b-861d-4139-bdde-2e7ed7e81420 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:31.217738 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7ab41b-861d-4139-bdde-2e7ed7e81420 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:31.218300 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7ab41b-861d-4139-bdde-2e7ed7e81420 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:31.218768 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7ab41b-861d-4139-bdde-2e7ed7e81420 admin admin] PATH_INFO: `/v3/projects/062e383759c34ca3b52c150dd337a6ba` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:31.220348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7ab41b-861d-4139-bdde-2e7ed7e81420 admin admin] RBAC: Authorizing `identity:get_project(project_id=062e383759c34ca3b52c150dd337a6ba)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:31.222643 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7ab41b-861d-4139-bdde-2e7ed7e81420 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:31.224862 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 150/295] 10.222.0.42 () {56 vars in 1207 bytes} [Thu Oct 27 01:11:31 2022] GET /identity/v3/projects/062e383759c34ca3b52c150dd337a6ba => generated 344 bytes in 25 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:31.230847 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbedd35f-fce9-4061-8d36-623bb30f909b admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:31.242659 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbedd35f-fce9-4061-8d36-623bb30f909b admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:31.247719 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10d9612-76c5-4f50-a90b-9bea2327533e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:31.248654 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10d9612-76c5-4f50-a90b-9bea2327533e admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:31.249252 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10d9612-76c5-4f50-a90b-9bea2327533e admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:31.249712 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10d9612-76c5-4f50-a90b-9bea2327533e admin admin] PATH_INFO: `/v3/projects/062e383759c34ca3b52c150dd337a6ba/groups/1bdbf8d0f4684cf59bcd5fb9b6ada058/roles/414286927fdd44688107ff5e10235acf` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:31.252075 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10d9612-76c5-4f50-a90b-9bea2327533e admin admin] RBAC: Authorizing `identity:create_grant(project_id=062e383759c34ca3b52c150dd337a6ba, group_id=1bdbf8d0f4684cf59bcd5fb9b6ada058, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:31.253449 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: WARNING py.warnings [None req-d10d9612-76c5-4f50-a90b-9bea2327533e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 27 01:11:31.253449 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: warnings.warn(msg) Oct 27 01:11:31.253449 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: Oct 27 01:11:31.256062 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10d9612-76c5-4f50-a90b-9bea2327533e admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:31.275197 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 146/296] 10.222.0.42 () {58 vars in 1463 bytes} [Thu Oct 27 01:11:31 2022] PUT /identity/v3/projects/062e383759c34ca3b52c150dd337a6ba/groups/1bdbf8d0f4684cf59bcd5fb9b6ada058/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 45 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:11:31.647805 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2112f35-2771-4e95-a518-d99bbb8c29f8 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:31.648318 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2112f35-2771-4e95-a518-d99bbb8c29f8 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:31.648392 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2112f35-2771-4e95-a518-d99bbb8c29f8 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:31.649926 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 151/297] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:31.657007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bb600f-d585-40de-a036-938fcf57dd7f None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:31.657328 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bb600f-d585-40de-a036-938fcf57dd7f None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:31.657596 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bb600f-d585-40de-a036-938fcf57dd7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:31.677916 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-21bb600f-d585-40de-a036-938fcf57dd7f None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:31.679142 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-21bb600f-d585-40de-a036-938fcf57dd7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:31.735564 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:31.760732 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 147/298] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:11:32.204312 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2112f35-2771-4e95-a518-d99bbb8c29f8 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:32.289162 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:32.349586 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2112f35-2771-4e95-a518-d99bbb8c29f8 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:32.396400 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f11b373-7b5e-443e-98d8-056aa093d1d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:32.397037 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f11b373-7b5e-443e-98d8-056aa093d1d8 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:32.397342 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f11b373-7b5e-443e-98d8-056aa093d1d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:32.397618 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f11b373-7b5e-443e-98d8-056aa093d1d8 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:32.409459 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0684b9-75e5-42e2-9971-13982b5f6aed None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:32.409941 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0684b9-75e5-42e2-9971-13982b5f6aed None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:32.410014 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0684b9-75e5-42e2-9971-13982b5f6aed None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:32.412117 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 148/299] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:32.413806 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f11b373-7b5e-443e-98d8-056aa093d1d8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:32.417406 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74163aa4-a5d5-42cb-82d4-d02f3fba6529 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:32.417683 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74163aa4-a5d5-42cb-82d4-d02f3fba6529 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:32.417949 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74163aa4-a5d5-42cb-82d4-d02f3fba6529 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:32.420214 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-2f11b373-7b5e-443e-98d8-056aa093d1d8 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 27 01:11:32.420214 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:32.420214 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:32.420214 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:32.420214 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:32.420214 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:32.420214 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:32.420214 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:32.420214 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:32.420214 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:32.420214 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:32.420214 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:32.420214 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:32.420214 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:32.420214 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:32.420214 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:32.421132 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:32.421132 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:32.421132 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:32.421132 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:32.421132 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:32.421132 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:32.421132 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:32.421132 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:32.421132 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:32.421132 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:32.421132 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:32.421132 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:32.421132 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:32.421132 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:32.421132 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:32.421132 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:32.423371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:32.423371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:32.423371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:32.423371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:32.423371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:32.423371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:32.423371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:32.423371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:32.423371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:32.423371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:32.423371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 27 01:11:32.423371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:32.423371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 152/300] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:11:32 2022] GET /identity/v3/roles/member => generated 84 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:32.427350 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f11b373-7b5e-443e-98d8-056aa093d1d8 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:32.456512 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-74163aa4-a5d5-42cb-82d4-d02f3fba6529 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:32.457833 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-74163aa4-a5d5-42cb-82d4-d02f3fba6529 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:32.547763 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f11b373-7b5e-443e-98d8-056aa093d1d8 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:32.549576 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 149/301] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 27 01:11:32.549928 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49471554-1e8e-4b16-8706-a07a6545a2a4 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:32.550308 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49471554-1e8e-4b16-8706-a07a6545a2a4 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:32.550679 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49471554-1e8e-4b16-8706-a07a6545a2a4 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:32.552786 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 150/302] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:32.558690 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2e5e05-cc1a-403d-bcb2-3462666770f3 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:32.559090 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2e5e05-cc1a-403d-bcb2-3462666770f3 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:32.559436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2e5e05-cc1a-403d-bcb2-3462666770f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:32.577043 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:32.579733 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-5a2e5e05-cc1a-403d-bcb2-3462666770f3 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:32.580603 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-5a2e5e05-cc1a-403d-bcb2-3462666770f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:32.593261 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390539b8-5d81-43c3-91e3-a4e697cb9e8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:32.594203 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390539b8-5d81-43c3-91e3-a4e697cb9e8a None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:32.594685 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390539b8-5d81-43c3-91e3-a4e697cb9e8a None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:32.595191 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390539b8-5d81-43c3-91e3-a4e697cb9e8a None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:32.596712 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-390539b8-5d81-43c3-91e3-a4e697cb9e8a None admin] RBAC: Adding query filter params (name=member) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:32.599202 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-390539b8-5d81-43c3-91e3-a4e697cb9e8a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:32.601723 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-390539b8-5d81-43c3-91e3-a4e697cb9e8a None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:32.619321 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 153/303] 10.222.0.42 () {56 vars in 1101 bytes} [Thu Oct 27 01:11:32 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:11:32.626367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390539b8-5d81-43c3-91e3-a4e697cb9e8a None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:32.702541 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 151/304] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 27 01:11:32.768968 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390539b8-5d81-43c3-91e3-a4e697cb9e8a None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:32.769603 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ce06f7-6173-4e71-8fc8-a31661be911d None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:32.770447 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ce06f7-6173-4e71-8fc8-a31661be911d None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:32.770724 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ce06f7-6173-4e71-8fc8-a31661be911d None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:32.772411 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 152/305] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 27 01:11:32.778285 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac71ff5-5a79-46d9-a0c7-19207fc415a2 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:32.778582 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac71ff5-5a79-46d9-a0c7-19207fc415a2 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:32.779015 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac71ff5-5a79-46d9-a0c7-19207fc415a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:32.810701 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc20b79-be7b-4469-af3c-c347fd67570d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:32.811671 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc20b79-be7b-4469-af3c-c347fd67570d None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:32.812173 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc20b79-be7b-4469-af3c-c347fd67570d None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:32.812654 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc20b79-be7b-4469-af3c-c347fd67570d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:32.822155 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-9ac71ff5-5a79-46d9-a0c7-19207fc415a2 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:32.823682 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-9ac71ff5-5a79-46d9-a0c7-19207fc415a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:32.876751 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:32.897015 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:32.911548 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc20b79-be7b-4469-af3c-c347fd67570d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:32.913783 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc20b79-be7b-4469-af3c-c347fd67570d None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:32.916492 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 153/306] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 27 01:11:32.917476 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac71ff5-5a79-46d9-a0c7-19207fc415a2 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:33.036714 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 154/307] 10.222.0.42 () {58 vars in 1282 bytes} [Thu Oct 27 01:11:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 411 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 27 01:11:33.043774 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc20b79-be7b-4469-af3c-c347fd67570d None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:33.075157 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac71ff5-5a79-46d9-a0c7-19207fc415a2 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:33.139733 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:33.140027 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1555ffb1-5a2f-45b5-a9df-798b5fc9e3cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:33.141278 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1555ffb1-5a2f-45b5-a9df-798b5fc9e3cc None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:33.141730 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1555ffb1-5a2f-45b5-a9df-798b5fc9e3cc None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:33.142137 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1555ffb1-5a2f-45b5-a9df-798b5fc9e3cc None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:33.157471 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc20b79-be7b-4469-af3c-c347fd67570d None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:33.164436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1555ffb1-5a2f-45b5-a9df-798b5fc9e3cc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:33.173992 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-1555ffb1-5a2f-45b5-a9df-798b5fc9e3cc None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 27 01:11:33.173992 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:33.173992 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:33.173992 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:33.173992 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:33.173992 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:33.173992 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:33.173992 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:33.173992 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:33.173992 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:33.173992 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:33.173992 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:33.173992 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:33.173992 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:33.173992 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:33.173992 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:33.175102 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:33.175102 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:33.175102 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:33.175102 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:33.175102 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:33.175102 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:33.175102 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:33.175102 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:33.175102 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:33.175102 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:33.175102 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:33.175102 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:33.175102 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:33.175102 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:33.175102 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:33.175102 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:33.176013 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:33.176013 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:33.176013 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:33.176013 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:33.176013 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:33.176013 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:33.176013 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:33.176013 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:33.176013 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:33.176013 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:33.176013 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 27 01:11:33.176013 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:33.178559 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 154/308] 10.222.0.42 () {56 vars in 1114 bytes} [Thu Oct 27 01:11:32 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 261 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:33.184217 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1555ffb1-5a2f-45b5-a9df-798b5fc9e3cc None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:33.201084 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42591dcc-9f84-4816-871c-1fa5330b2cc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:33.201084 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42591dcc-9f84-4816-871c-1fa5330b2cc8 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:33.201084 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42591dcc-9f84-4816-871c-1fa5330b2cc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:33.201084 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42591dcc-9f84-4816-871c-1fa5330b2cc8 None admin] PATH_INFO: `/v3/projects/50a39f634ddf41798003dd395f14be68` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:33.201506 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42591dcc-9f84-4816-871c-1fa5330b2cc8 None admin] RBAC: Authorizing `identity:get_project(project_id=50a39f634ddf41798003dd395f14be68)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:33.201506 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42591dcc-9f84-4816-871c-1fa5330b2cc8 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:33.207082 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 155/309] 10.222.0.42 () {56 vars in 1186 bytes} [Thu Oct 27 01:11:33 2022] GET /identity/v3/projects/50a39f634ddf41798003dd395f14be68 => generated 298 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:33.208734 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42591dcc-9f84-4816-871c-1fa5330b2cc8 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:33.242272 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:33.330379 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42591dcc-9f84-4816-871c-1fa5330b2cc8 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:33.334175 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1555ffb1-5a2f-45b5-a9df-798b5fc9e3cc None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:33.369277 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b213bc-1432-4406-859b-3fd863f7bd93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:33.369908 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b213bc-1432-4406-859b-3fd863f7bd93 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:33.370249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b213bc-1432-4406-859b-3fd863f7bd93 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:33.370460 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b213bc-1432-4406-859b-3fd863f7bd93 None admin] PATH_INFO: `/v3/groups/3215d70f63f84c98a348daded668804e` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:33.371624 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29b213bc-1432-4406-859b-3fd863f7bd93 None admin] RBAC: Authorizing `identity:get_group(group_id=3215d70f63f84c98a348daded668804e)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:33.373177 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29b213bc-1432-4406-859b-3fd863f7bd93 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:33.373301 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f586b9c-7f83-4ac0-9178-acf0fa357e40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:33.373847 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f586b9c-7f83-4ac0-9178-acf0fa357e40 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:33.374134 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f586b9c-7f83-4ac0-9178-acf0fa357e40 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:33.374408 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f586b9c-7f83-4ac0-9178-acf0fa357e40 None admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:33.375063 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 156/310] 10.222.0.42 () {56 vars in 1180 bytes} [Thu Oct 27 01:11:33 2022] GET /identity/v3/groups/3215d70f63f84c98a348daded668804e => generated 226 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:33.375845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b213bc-1432-4406-859b-3fd863f7bd93 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:33.378799 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f586b9c-7f83-4ac0-9178-acf0fa357e40 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:33.380706 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f586b9c-7f83-4ac0-9178-acf0fa357e40 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:33.382422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f586b9c-7f83-4ac0-9178-acf0fa357e40 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:33.397770 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 155/311] 10.222.0.42 () {56 vars in 1111 bytes} [Thu Oct 27 01:11:33 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:33.398631 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f586b9c-7f83-4ac0-9178-acf0fa357e40 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:33.407133 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:33.511652 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b213bc-1432-4406-859b-3fd863f7bd93 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:33.515041 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:33.524301 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f586b9c-7f83-4ac0-9178-acf0fa357e40 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:33.554669 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565541cf-f37f-4a78-bf23-9a3b889f3ca4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:33.555924 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565541cf-f37f-4a78-bf23-9a3b889f3ca4 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:33.556685 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565541cf-f37f-4a78-bf23-9a3b889f3ca4 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:33.557214 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565541cf-f37f-4a78-bf23-9a3b889f3ca4 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:33.565156 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f774d41-3716-43bb-b745-10ebe53e7486 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:33.565857 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f774d41-3716-43bb-b745-10ebe53e7486 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:33.566147 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f774d41-3716-43bb-b745-10ebe53e7486 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:33.566401 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f774d41-3716-43bb-b745-10ebe53e7486 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:33.570575 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-565541cf-f37f-4a78-bf23-9a3b889f3ca4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:33.576288 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-565541cf-f37f-4a78-bf23-9a3b889f3ca4 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 27 01:11:33.576288 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:33.576288 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:33.576288 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:33.576288 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:33.576288 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:33.576288 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:33.576288 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:33.576288 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:33.576288 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:33.576288 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:33.576288 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:33.576288 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:33.576288 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:33.576288 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:33.576288 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:33.577441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:33.577441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:33.577441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:33.577441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:33.577441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:33.577441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:33.577441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:33.577441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:33.577441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:33.577441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:33.577441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:33.577441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:33.577441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:33.577441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:33.577441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:33.577441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:33.578423 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:33.578423 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:33.578423 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:33.578423 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:33.578423 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:33.578423 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:33.578423 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:33.578423 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:33.578423 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:33.578423 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:33.578423 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 27 01:11:33.578423 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:33.578423 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 157/312] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:11:33 2022] GET /identity/v3/roles/member => generated 84 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:33.579209 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565541cf-f37f-4a78-bf23-9a3b889f3ca4 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:33.583978 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f774d41-3716-43bb-b745-10ebe53e7486 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:33.589521 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-4f774d41-3716-43bb-b745-10ebe53e7486 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 27 01:11:33.589521 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:33.589521 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:33.589521 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:33.589521 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:33.589521 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:33.589521 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:33.589521 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:33.589521 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:33.589521 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:33.589521 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:33.589521 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:33.589521 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:33.589521 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:33.589521 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:33.589521 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:33.590558 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:33.590558 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:33.590558 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:33.590558 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:33.590558 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:33.590558 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:33.590558 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:33.590558 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:33.590558 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:33.590558 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:33.590558 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:33.590558 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:33.590558 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:33.590558 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:33.590558 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:33.590558 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:33.591439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:33.591439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:33.591439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:33.591439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:33.591439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:33.591439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:33.591439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:33.591439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:33.591439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:33.591439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:33.591439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 27 01:11:33.591439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:33.594340 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 156/313] 10.222.0.42 () {56 vars in 1114 bytes} [Thu Oct 27 01:11:33 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:33.595076 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f774d41-3716-43bb-b745-10ebe53e7486 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:33.690841 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:33.721230 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565541cf-f37f-4a78-bf23-9a3b889f3ca4 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:33.729332 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f774d41-3716-43bb-b745-10ebe53e7486 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:33.768184 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfdf7d08-da74-4c0a-bd8d-a3b242dc955a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:33.768803 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdf7d08-da74-4c0a-bd8d-a3b242dc955a None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:33.769104 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdf7d08-da74-4c0a-bd8d-a3b242dc955a None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:33.769609 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdf7d08-da74-4c0a-bd8d-a3b242dc955a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:33.770655 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfdf7d08-da74-4c0a-bd8d-a3b242dc955a None admin] RBAC: Adding query filter params (group.id=3215d70f63f84c98a348daded668804e, role.id=f199b79a7178462fa67bb8243943408f, scope.project.id=50a39f634ddf41798003dd395f14be68) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:33.771139 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfdf7d08-da74-4c0a-bd8d-a3b242dc955a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:33.772602 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfdf7d08-da74-4c0a-bd8d-a3b242dc955a None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:33.773486 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef71d0b-890c-44ae-a634-726035aa36fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:33.774612 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef71d0b-890c-44ae-a634-726035aa36fc None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:33.775106 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef71d0b-890c-44ae-a634-726035aa36fc None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:33.776221 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef71d0b-890c-44ae-a634-726035aa36fc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:33.779214 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 158/314] 10.222.0.42 () {56 vars in 1376 bytes} [Thu Oct 27 01:11:33 2022] GET /identity/v3/role_assignments?group.id=3215d70f63f84c98a348daded668804e&scope.project.id=50a39f634ddf41798003dd395f14be68&role.id=f199b79a7178462fa67bb8243943408f => generated 611 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:33.780682 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfdf7d08-da74-4c0a-bd8d-a3b242dc955a None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:33.805262 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:33.876058 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef71d0b-890c-44ae-a634-726035aa36fc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:33.877499 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef71d0b-890c-44ae-a634-726035aa36fc None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:33.946907 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfdf7d08-da74-4c0a-bd8d-a3b242dc955a None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:33.989810 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53ee1f7-dd22-4518-a7dd-bb18e1e7e3c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:33.990439 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53ee1f7-dd22-4518-a7dd-bb18e1e7e3c5 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:33.990716 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53ee1f7-dd22-4518-a7dd-bb18e1e7e3c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:33.990988 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53ee1f7-dd22-4518-a7dd-bb18e1e7e3c5 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:33.991662 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53ee1f7-dd22-4518-a7dd-bb18e1e7e3c5 None admin] RBAC: Adding query filter params (name=member) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:33.992718 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53ee1f7-dd22-4518-a7dd-bb18e1e7e3c5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:33.994722 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53ee1f7-dd22-4518-a7dd-bb18e1e7e3c5 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:34.006882 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 159/315] 10.222.0.42 () {56 vars in 1101 bytes} [Thu Oct 27 01:11:33 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:34.007108 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 157/316] 10.222.0.42 () {58 vars in 1282 bytes} [Thu Oct 27 01:11:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 412 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 27 01:11:34.007343 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53ee1f7-dd22-4518-a7dd-bb18e1e7e3c5 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:34.012301 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef71d0b-890c-44ae-a634-726035aa36fc None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:34.023327 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:34.092303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:34.142473 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53ee1f7-dd22-4518-a7dd-bb18e1e7e3c5 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:34.148454 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef71d0b-890c-44ae-a634-726035aa36fc None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:34.186312 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8cb0ef-4b5a-4822-8b5e-8b82d7a35502 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:34.186919 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8cb0ef-4b5a-4822-8b5e-8b82d7a35502 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:34.187194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8cb0ef-4b5a-4822-8b5e-8b82d7a35502 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:34.187463 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8cb0ef-4b5a-4822-8b5e-8b82d7a35502 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:34.188124 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8cb0ef-4b5a-4822-8b5e-8b82d7a35502 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:34.189228 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8cb0ef-4b5a-4822-8b5e-8b82d7a35502 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:34.190561 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eea25e2-d42d-48e5-b060-15cf811df9d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:34.190683 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8cb0ef-4b5a-4822-8b5e-8b82d7a35502 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:34.191529 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eea25e2-d42d-48e5-b060-15cf811df9d9 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:34.192157 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eea25e2-d42d-48e5-b060-15cf811df9d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:34.192639 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eea25e2-d42d-48e5-b060-15cf811df9d9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:34.202725 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 160/317] 10.222.0.42 () {56 vars in 1111 bytes} [Thu Oct 27 01:11:34 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:34.203734 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8cb0ef-4b5a-4822-8b5e-8b82d7a35502 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:34.282698 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eea25e2-d42d-48e5-b060-15cf811df9d9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:34.284356 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eea25e2-d42d-48e5-b060-15cf811df9d9 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:34.286613 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:34.332817 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8cb0ef-4b5a-4822-8b5e-8b82d7a35502 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:34.351283 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:34.379718 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-834e8246-74c2-4d7b-8192-17e32b2c11fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:34.380357 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834e8246-74c2-4d7b-8192-17e32b2c11fa None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:34.380635 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834e8246-74c2-4d7b-8192-17e32b2c11fa None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:34.380912 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834e8246-74c2-4d7b-8192-17e32b2c11fa None admin] PATH_INFO: `/v3/projects/50a39f634ddf41798003dd395f14be68` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:34.382080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-834e8246-74c2-4d7b-8192-17e32b2c11fa None admin] RBAC: Authorizing `identity:get_project(project_id=50a39f634ddf41798003dd395f14be68)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:34.383538 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-834e8246-74c2-4d7b-8192-17e32b2c11fa None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:34.386899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 161/318] 10.222.0.42 () {56 vars in 1186 bytes} [Thu Oct 27 01:11:34 2022] GET /identity/v3/projects/50a39f634ddf41798003dd395f14be68 => generated 298 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:34.387877 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-834e8246-74c2-4d7b-8192-17e32b2c11fa None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:34.407585 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 158/319] 10.222.0.42 () {58 vars in 1282 bytes} [Thu Oct 27 01:11:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 396 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 27 01:11:34.408457 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eea25e2-d42d-48e5-b060-15cf811df9d9 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:34.524600 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-834e8246-74c2-4d7b-8192-17e32b2c11fa None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:34.532599 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eea25e2-d42d-48e5-b060-15cf811df9d9 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:34.564977 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d92ee8-4287-40c8-82aa-cfa00c389a43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:34.565643 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d92ee8-4287-40c8-82aa-cfa00c389a43 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:34.565922 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d92ee8-4287-40c8-82aa-cfa00c389a43 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:34.566191 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d92ee8-4287-40c8-82aa-cfa00c389a43 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:34.570344 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:34.577206 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c8d70c-8d86-4b2b-b1f4-66781c76879d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:34.578105 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c8d70c-8d86-4b2b-b1f4-66781c76879d None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:34.578585 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c8d70c-8d86-4b2b-b1f4-66781c76879d None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:34.579054 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c8d70c-8d86-4b2b-b1f4-66781c76879d None admin] PATH_INFO: `/v3/groups/3215d70f63f84c98a348daded668804e` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:34.580420 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8c8d70c-8d86-4b2b-b1f4-66781c76879d None admin] RBAC: Authorizing `identity:get_group(group_id=3215d70f63f84c98a348daded668804e)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:34.582006 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8c8d70c-8d86-4b2b-b1f4-66781c76879d None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:34.584239 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 159/320] 10.222.0.42 () {56 vars in 1180 bytes} [Thu Oct 27 01:11:34 2022] GET /identity/v3/groups/3215d70f63f84c98a348daded668804e => generated 226 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:34.585010 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c8d70c-8d86-4b2b-b1f4-66781c76879d None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:34.614870 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:34.656685 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29d92ee8-4287-40c8-82aa-cfa00c389a43 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:34.658316 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29d92ee8-4287-40c8-82aa-cfa00c389a43 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:34.710354 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c8d70c-8d86-4b2b-b1f4-66781c76879d None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:34.750122 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7a4b05-72cf-4744-bc3d-eaaa2326f488 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:34.750756 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7a4b05-72cf-4744-bc3d-eaaa2326f488 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:34.751030 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7a4b05-72cf-4744-bc3d-eaaa2326f488 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:34.751290 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7a4b05-72cf-4744-bc3d-eaaa2326f488 None admin] PATH_INFO: `/v3/projects/69bfdc060ff949e2ba795cad23f9f746` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:34.752390 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7a4b05-72cf-4744-bc3d-eaaa2326f488 None admin] RBAC: Authorizing `identity:get_project(project_id=69bfdc060ff949e2ba795cad23f9f746)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:34.753857 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7a4b05-72cf-4744-bc3d-eaaa2326f488 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:34.756664 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 160/321] 10.222.0.42 () {56 vars in 1186 bytes} [Thu Oct 27 01:11:34 2022] GET /identity/v3/projects/69bfdc060ff949e2ba795cad23f9f746 => generated 302 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:34.757792 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e285081b-5816-4aca-b716-56588944171e None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:34.758067 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e285081b-5816-4aca-b716-56588944171e None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:34.758329 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e285081b-5816-4aca-b716-56588944171e None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:34.759501 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 161/322] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:34 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:34.760180 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e285081b-5816-4aca-b716-56588944171e None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:34.783306 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 162/323] 10.222.0.42 () {58 vars in 1282 bytes} [Thu Oct 27 01:11:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 396 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 27 01:11:34.784896 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d92ee8-4287-40c8-82aa-cfa00c389a43 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:34.821984 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:34.855053 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:34.865885 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e285081b-5816-4aca-b716-56588944171e None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:34.886322 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d92ee8-4287-40c8-82aa-cfa00c389a43 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:34.900941 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab64506-a2b3-4892-a313-9e3eeeb9c61e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:34.901570 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab64506-a2b3-4892-a313-9e3eeeb9c61e None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:34.901848 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab64506-a2b3-4892-a313-9e3eeeb9c61e None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:34.902117 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab64506-a2b3-4892-a313-9e3eeeb9c61e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:34.902835 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab64506-a2b3-4892-a313-9e3eeeb9c61e None admin] RBAC: Adding query filter params (group.id=3215d70f63f84c98a348daded668804e, role.id=bb9bd8ef24ff4aa5b639a2bd8567345b, scope.project.id=50a39f634ddf41798003dd395f14be68) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:34.903142 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab64506-a2b3-4892-a313-9e3eeeb9c61e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:34.904565 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab64506-a2b3-4892-a313-9e3eeeb9c61e None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:34.910209 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 162/324] 10.222.0.42 () {56 vars in 1376 bytes} [Thu Oct 27 01:11:34 2022] GET /identity/v3/role_assignments?group.id=3215d70f63f84c98a348daded668804e&scope.project.id=50a39f634ddf41798003dd395f14be68&role.id=bb9bd8ef24ff4aa5b639a2bd8567345b => generated 611 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:34.912063 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305011a1-d9ae-4384-8f83-2b8b6980489f None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:34.912346 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305011a1-d9ae-4384-8f83-2b8b6980489f None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:34.912619 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305011a1-d9ae-4384-8f83-2b8b6980489f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:34.932126 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-305011a1-d9ae-4384-8f83-2b8b6980489f None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:34.932507 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16bcc238-0c68-47b8-85d9-a47bbbe8b612 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:34.933125 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16bcc238-0c68-47b8-85d9-a47bbbe8b612 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:34.933410 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16bcc238-0c68-47b8-85d9-a47bbbe8b612 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:34.933505 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-305011a1-d9ae-4384-8f83-2b8b6980489f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:34.933680 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16bcc238-0c68-47b8-85d9-a47bbbe8b612 None admin] PATH_INFO: `/v3/groups/3215d70f63f84c98a348daded668804e` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:34.934855 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16bcc238-0c68-47b8-85d9-a47bbbe8b612 None admin] RBAC: Authorizing `identity:get_group(group_id=3215d70f63f84c98a348daded668804e)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:34.936311 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16bcc238-0c68-47b8-85d9-a47bbbe8b612 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:34.938166 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 163/325] 10.222.0.42 () {56 vars in 1180 bytes} [Thu Oct 27 01:11:34 2022] GET /identity/v3/groups/3215d70f63f84c98a348daded668804e => generated 226 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:34.944181 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16bcc238-0c68-47b8-85d9-a47bbbe8b612 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:35.008956 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 163/326] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:11:35.013457 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305011a1-d9ae-4384-8f83-2b8b6980489f None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:35.063872 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16bcc238-0c68-47b8-85d9-a47bbbe8b612 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:35.067946 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:35.108246 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1a40ca-44f8-4caa-9322-cd372a57bae4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:35.110912 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1a40ca-44f8-4caa-9322-cd372a57bae4 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:35.110912 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1a40ca-44f8-4caa-9322-cd372a57bae4 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:35.110912 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1a40ca-44f8-4caa-9322-cd372a57bae4 None admin] PATH_INFO: `/v3/projects/69bfdc060ff949e2ba795cad23f9f746` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:35.110912 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf1a40ca-44f8-4caa-9322-cd372a57bae4 None admin] RBAC: Authorizing `identity:get_project(project_id=69bfdc060ff949e2ba795cad23f9f746)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:35.112015 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf1a40ca-44f8-4caa-9322-cd372a57bae4 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:35.114350 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 164/327] 10.222.0.42 () {56 vars in 1186 bytes} [Thu Oct 27 01:11:34 2022] GET /identity/v3/projects/69bfdc060ff949e2ba795cad23f9f746 => generated 302 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:35.119301 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1a40ca-44f8-4caa-9322-cd372a57bae4 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:35.130041 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:35.148158 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305011a1-d9ae-4384-8f83-2b8b6980489f None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:35.194822 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed106575-dc5f-4f67-b18f-e2368d012af4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:35.195673 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed106575-dc5f-4f67-b18f-e2368d012af4 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:35.196175 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed106575-dc5f-4f67-b18f-e2368d012af4 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:35.196682 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed106575-dc5f-4f67-b18f-e2368d012af4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:35.197744 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed106575-dc5f-4f67-b18f-e2368d012af4 None admin] RBAC: Adding query filter params (group.id=3215d70f63f84c98a348daded668804e, role.id=f199b79a7178462fa67bb8243943408f, scope.project.id=69bfdc060ff949e2ba795cad23f9f746) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:35.198253 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed106575-dc5f-4f67-b18f-e2368d012af4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:35.200118 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed106575-dc5f-4f67-b18f-e2368d012af4 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:35.211904 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 164/328] 10.222.0.42 () {56 vars in 1376 bytes} [Thu Oct 27 01:11:35 2022] GET /identity/v3/role_assignments?group.id=3215d70f63f84c98a348daded668804e&scope.project.id=69bfdc060ff949e2ba795cad23f9f746&role.id=f199b79a7178462fa67bb8243943408f => generated 611 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:35.243569 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1a40ca-44f8-4caa-9322-cd372a57bae4 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:35.282634 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d2499d-5d32-42ec-9483-b9a806eef012 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:35.283331 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d2499d-5d32-42ec-9483-b9a806eef012 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:35.283606 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d2499d-5d32-42ec-9483-b9a806eef012 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:35.283871 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d2499d-5d32-42ec-9483-b9a806eef012 None admin] PATH_INFO: `/v3/groups/3215d70f63f84c98a348daded668804e` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:35.285016 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d2499d-5d32-42ec-9483-b9a806eef012 None admin] RBAC: Authorizing `identity:get_group(group_id=3215d70f63f84c98a348daded668804e)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:35.286828 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d2499d-5d32-42ec-9483-b9a806eef012 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:35.288771 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 165/329] 10.222.0.42 () {56 vars in 1180 bytes} [Thu Oct 27 01:11:35 2022] GET /identity/v3/groups/3215d70f63f84c98a348daded668804e => generated 226 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:11:35.294765 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed106575-dc5f-4f67-b18f-e2368d012af4 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:35.295552 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d2499d-5d32-42ec-9483-b9a806eef012 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:35.347058 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:35.432142 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d2499d-5d32-42ec-9483-b9a806eef012 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:35.439394 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed106575-dc5f-4f67-b18f-e2368d012af4 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:35.472456 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10316ac-1386-4fb0-9aea-12df65389f12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:35.473581 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10316ac-1386-4fb0-9aea-12df65389f12 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:35.474483 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10316ac-1386-4fb0-9aea-12df65389f12 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:35.474796 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10316ac-1386-4fb0-9aea-12df65389f12 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:35.481787 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5799f518-140a-47f6-8249-300e6218c5e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:35.482398 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5799f518-140a-47f6-8249-300e6218c5e3 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:35.482680 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5799f518-140a-47f6-8249-300e6218c5e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:35.482957 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5799f518-140a-47f6-8249-300e6218c5e3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:35.483689 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5799f518-140a-47f6-8249-300e6218c5e3 None admin] RBAC: Adding query filter params (group.id=3215d70f63f84c98a348daded668804e, role.id=bb9bd8ef24ff4aa5b639a2bd8567345b, scope.project.id=69bfdc060ff949e2ba795cad23f9f746) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:35.483987 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5799f518-140a-47f6-8249-300e6218c5e3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:35.485474 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5799f518-140a-47f6-8249-300e6218c5e3 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:35.487420 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f10316ac-1386-4fb0-9aea-12df65389f12 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:35.492777 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 165/330] 10.222.0.42 () {56 vars in 1376 bytes} [Thu Oct 27 01:11:35 2022] GET /identity/v3/role_assignments?group.id=3215d70f63f84c98a348daded668804e&scope.project.id=69bfdc060ff949e2ba795cad23f9f746&role.id=bb9bd8ef24ff4aa5b639a2bd8567345b => generated 611 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:35.493041 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-f10316ac-1386-4fb0-9aea-12df65389f12 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:11:35.493041 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:35.493041 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:35.493041 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:35.493041 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:35.493041 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:35.493041 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:35.493041 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:35.493041 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:35.493041 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:35.493041 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:35.493041 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:35.493041 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:35.493041 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:35.493041 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:35.493041 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:35.493962 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:35.493962 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:35.493962 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:35.493962 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:35.493962 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:35.493962 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:35.493962 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:35.493962 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:35.493962 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:35.493962 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:35.493962 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:35.493962 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:35.493962 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:35.493962 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:35.493962 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:35.493962 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:35.494771 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:35.494771 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:35.494771 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:35.494771 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:35.494771 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:35.494771 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:35.494771 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:35.494771 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:35.494771 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:35.494771 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:35.494771 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:11:35.494771 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:35.494771 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 166/331] 10.222.0.42 () {56 vars in 1096 bytes} [Thu Oct 27 01:11:35 2022] GET /identity/v3/roles/admin => generated 83 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:35.501470 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5799f518-140a-47f6-8249-300e6218c5e3 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:35.519501 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:35.622671 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5799f518-140a-47f6-8249-300e6218c5e3 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:35.663425 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112ca54b-4c5f-474f-9523-a84f68adc0d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:35.664416 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112ca54b-4c5f-474f-9523-a84f68adc0d1 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:35.664692 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112ca54b-4c5f-474f-9523-a84f68adc0d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:35.664965 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112ca54b-4c5f-474f-9523-a84f68adc0d1 None admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:35.665698 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112ca54b-4c5f-474f-9523-a84f68adc0d1 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:35.666002 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112ca54b-4c5f-474f-9523-a84f68adc0d1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:35.667375 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112ca54b-4c5f-474f-9523-a84f68adc0d1 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:35.676307 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 166/332] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:11:35 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:35.682401 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10316ac-1386-4fb0-9aea-12df65389f12 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:35.786402 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:35.789579 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10316ac-1386-4fb0-9aea-12df65389f12 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:35.826038 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a02a22-514d-47e7-9dcb-7cc657cacb12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:35.826645 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a02a22-514d-47e7-9dcb-7cc657cacb12 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:35.826922 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a02a22-514d-47e7-9dcb-7cc657cacb12 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:35.827216 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a02a22-514d-47e7-9dcb-7cc657cacb12 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:35.897742 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a02a22-514d-47e7-9dcb-7cc657cacb12 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:35.899196 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a02a22-514d-47e7-9dcb-7cc657cacb12 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:35.994844 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:35.998238 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 167/333] 10.222.0.42 () {58 vars in 1282 bytes} [Thu Oct 27 01:11:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 27 01:11:36.003903 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112ca54b-4c5f-474f-9523-a84f68adc0d1 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:36.077547 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:36.101854 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112ca54b-4c5f-474f-9523-a84f68adc0d1 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:36.134142 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8988cf-8e42-44f3-82cb-858f90a9bf60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:36.134761 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8988cf-8e42-44f3-82cb-858f90a9bf60 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:36.135038 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8988cf-8e42-44f3-82cb-858f90a9bf60 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:36.135308 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8988cf-8e42-44f3-82cb-858f90a9bf60 None admin] PATH_INFO: `/v3/projects/062e383759c34ca3b52c150dd337a6ba` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:36.136417 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8988cf-8e42-44f3-82cb-858f90a9bf60 None admin] RBAC: Authorizing `identity:get_project(project_id=062e383759c34ca3b52c150dd337a6ba)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:36.137867 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8988cf-8e42-44f3-82cb-858f90a9bf60 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:36.139457 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 167/334] 10.222.0.42 () {56 vars in 1186 bytes} [Thu Oct 27 01:11:36 2022] GET /identity/v3/projects/062e383759c34ca3b52c150dd337a6ba => generated 344 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:36.144916 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a02a22-514d-47e7-9dcb-7cc657cacb12 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:36.239651 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a02a22-514d-47e7-9dcb-7cc657cacb12 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:36.272376 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dbd4638-815f-4268-b3d2-09dedc3040e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:36.272982 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbd4638-815f-4268-b3d2-09dedc3040e8 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:36.273287 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbd4638-815f-4268-b3d2-09dedc3040e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:36.273557 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbd4638-815f-4268-b3d2-09dedc3040e8 None admin] PATH_INFO: `/v3/groups/1bdbf8d0f4684cf59bcd5fb9b6ada058` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:36.274651 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dbd4638-815f-4268-b3d2-09dedc3040e8 None admin] RBAC: Authorizing `identity:get_group(group_id=1bdbf8d0f4684cf59bcd5fb9b6ada058)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:36.276082 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dbd4638-815f-4268-b3d2-09dedc3040e8 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:36.277645 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 168/335] 10.222.0.42 () {56 vars in 1180 bytes} [Thu Oct 27 01:11:36 2022] GET /identity/v3/groups/1bdbf8d0f4684cf59bcd5fb9b6ada058 => generated 229 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:36.283227 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8988cf-8e42-44f3-82cb-858f90a9bf60 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:36.378837 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8988cf-8e42-44f3-82cb-858f90a9bf60 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:36.411361 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24a9238-077d-4549-ae35-9b941ae7031e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:36.411949 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24a9238-077d-4549-ae35-9b941ae7031e None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:36.412226 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24a9238-077d-4549-ae35-9b941ae7031e None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:36.412496 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24a9238-077d-4549-ae35-9b941ae7031e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:36.413235 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24a9238-077d-4549-ae35-9b941ae7031e None admin] RBAC: Adding query filter params (group.id=1bdbf8d0f4684cf59bcd5fb9b6ada058, role.id=414286927fdd44688107ff5e10235acf, scope.project.id=062e383759c34ca3b52c150dd337a6ba) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:36.413540 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24a9238-077d-4549-ae35-9b941ae7031e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:36.414948 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24a9238-077d-4549-ae35-9b941ae7031e None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:36.420483 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 168/336] 10.222.0.42 () {56 vars in 1376 bytes} [Thu Oct 27 01:11:36 2022] GET /identity/v3/role_assignments?group.id=1bdbf8d0f4684cf59bcd5fb9b6ada058&scope.project.id=062e383759c34ca3b52c150dd337a6ba&role.id=414286927fdd44688107ff5e10235acf => generated 611 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:38.994849 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ee2ff0-6f7d-4bdb-9c56-b9329989e535 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:38.996244 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ee2ff0-6f7d-4bdb-9c56-b9329989e535 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:38.996737 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ee2ff0-6f7d-4bdb-9c56-b9329989e535 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:38.998461 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 169/337] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:38 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 27 01:11:39.011570 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31d8ee7-8696-43e3-895f-ef240f86dd04 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:39.012601 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31d8ee7-8696-43e3-895f-ef240f86dd04 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:39.013120 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31d8ee7-8696-43e3-895f-ef240f86dd04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:39.031174 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:39.042058 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-f31d8ee7-8696-43e3-895f-ef240f86dd04 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:39.043651 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-f31d8ee7-8696-43e3-895f-ef240f86dd04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:39.098641 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e8ccdd-ed24-468a-8500-d52109d59a5f None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:39.099235 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e8ccdd-ed24-468a-8500-d52109d59a5f None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:39.099714 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e8ccdd-ed24-468a-8500-d52109d59a5f None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:39.102740 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 170/338] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:39.107015 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13667cec-41b9-434d-9651-8869dccc52a1 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:39.107523 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13667cec-41b9-434d-9651-8869dccc52a1 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:39.107995 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13667cec-41b9-434d-9651-8869dccc52a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:39.138229 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-13667cec-41b9-434d-9651-8869dccc52a1 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:39.139411 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-13667cec-41b9-434d-9651-8869dccc52a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:39.143610 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 169/339] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:11:39.230970 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:39.238685 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 171/340] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 27 01:11:39.515280 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f31d8ee7-8696-43e3-895f-ef240f86dd04 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:39.530030 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e214b1d9-9769-431a-be33-998527c2f4da None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:39.530649 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e214b1d9-9769-431a-be33-998527c2f4da None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:39.531207 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e214b1d9-9769-431a-be33-998527c2f4da None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:39.532726 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 172/341] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 27 01:11:39.553086 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9294a8b1-674a-413a-b497-98fcc82ade88 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:39.553086 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9294a8b1-674a-413a-b497-98fcc82ade88 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:39.553086 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9294a8b1-674a-413a-b497-98fcc82ade88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:39.594092 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-9294a8b1-674a-413a-b497-98fcc82ade88 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:39.599335 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-9294a8b1-674a-413a-b497-98fcc82ade88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:39.623147 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f31d8ee7-8696-43e3-895f-ef240f86dd04 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:39.662211 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb000336-629f-4a38-af9a-93291403d145 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:39.663228 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb000336-629f-4a38-af9a-93291403d145 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:39.663766 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb000336-629f-4a38-af9a-93291403d145 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:39.664262 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb000336-629f-4a38-af9a-93291403d145 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:39.671638 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:39.684968 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb000336-629f-4a38-af9a-93291403d145 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:39.686767 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb000336-629f-4a38-af9a-93291403d145 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:39.709289 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-eb000336-629f-4a38-af9a-93291403d145 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:39.709289 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:39.709289 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:39.709289 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:39.709289 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:39.709289 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:39.709289 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:39.709289 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:39.709289 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:39.709289 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:39.709289 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:39.709289 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:39.709289 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:11:39.709289 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:11:39.709289 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:11:39.709289 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:11:39.710300 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:39.710300 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:39.710300 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:39.710300 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:39.710300 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:39.710300 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:39.710300 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:39.710300 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:39.710300 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:39.710300 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:39.710300 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:39.710300 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:39.710300 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:39.710300 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:39.710300 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:39.710300 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:39.710300 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:11:39.711175 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:11:39.711175 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:39.711175 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:39.711175 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 170/342] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:11:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:39.711430 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 173/343] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:11:39.716323 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb000336-629f-4a38-af9a-93291403d145 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:39.750731 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9294a8b1-674a-413a-b497-98fcc82ade88 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:39.843920 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb000336-629f-4a38-af9a-93291403d145 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:39.880024 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b4e007-3c70-4e7c-b748-9787b786adf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:39.880995 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b4e007-3c70-4e7c-b748-9787b786adf5 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:39.881620 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b4e007-3c70-4e7c-b748-9787b786adf5 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:39.882100 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b4e007-3c70-4e7c-b748-9787b786adf5 None admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:39.882944 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b4e007-3c70-4e7c-b748-9787b786adf5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:39.883447 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b4e007-3c70-4e7c-b748-9787b786adf5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:39.885004 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b4e007-3c70-4e7c-b748-9787b786adf5 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:39.898290 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 171/344] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:11:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:39.914206 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b4e007-3c70-4e7c-b748-9787b786adf5 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:39.975843 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9294a8b1-674a-413a-b497-98fcc82ade88 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:39.983326 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:40.019216 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:40.022186 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b4e007-3c70-4e7c-b748-9787b786adf5 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:40.029551 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98208681-cd59-4418-8fcd-321e28965282 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:40.030518 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98208681-cd59-4418-8fcd-321e28965282 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:40.030984 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98208681-cd59-4418-8fcd-321e28965282 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:40.031438 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98208681-cd59-4418-8fcd-321e28965282 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:40.042755 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98208681-cd59-4418-8fcd-321e28965282 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:40.044542 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98208681-cd59-4418-8fcd-321e28965282 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:40.059143 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200461d7-ab65-4ddf-8116-a642738f1558 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:40.059827 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200461d7-ab65-4ddf-8116-a642738f1558 None admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:40.060108 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200461d7-ab65-4ddf-8116-a642738f1558 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:40.060379 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200461d7-ab65-4ddf-8116-a642738f1558 None admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:40.061044 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-200461d7-ab65-4ddf-8116-a642738f1558 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:40.062551 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-200461d7-ab65-4ddf-8116-a642738f1558 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:40.063285 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-98208681-cd59-4418-8fcd-321e28965282 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:40.063285 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:40.063285 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:40.063285 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:40.063285 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:40.063285 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:40.063285 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:40.063285 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:40.063285 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:40.063285 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:40.063285 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:40.063285 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:40.063285 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:11:40.063285 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:11:40.063285 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:11:40.063285 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:11:40.064139 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:40.064139 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:40.064139 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:40.064139 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:40.064139 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:40.064139 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:40.064139 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:40.064139 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:40.064139 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:40.064139 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:40.064139 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:40.064139 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:40.064139 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:40.064139 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:40.064139 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:40.064139 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:40.064139 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:11:40.065083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:11:40.065083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:40.065083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:40.067614 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 174/345] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:11:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 317 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 27 01:11:40.068757 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98208681-cd59-4418-8fcd-321e28965282 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:40.095500 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 172/346] 10.222.0.42 () {60 vars in 1132 bytes} [Thu Oct 27 01:11:39 2022] POST /identity/v3/users => generated 246 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:11:40.096522 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200461d7-ab65-4ddf-8116-a642738f1558 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:40.222452 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200461d7-ab65-4ddf-8116-a642738f1558 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:40.225340 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:40.266804 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d959b4-7026-4bdc-a2ff-05a6a0c5378e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:40.267470 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d959b4-7026-4bdc-a2ff-05a6a0c5378e None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:40.267813 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d959b4-7026-4bdc-a2ff-05a6a0c5378e None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:40.268084 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d959b4-7026-4bdc-a2ff-05a6a0c5378e None admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:40.268789 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d959b4-7026-4bdc-a2ff-05a6a0c5378e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:40.269116 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d959b4-7026-4bdc-a2ff-05a6a0c5378e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:40.271091 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98208681-cd59-4418-8fcd-321e28965282 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:40.271341 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d959b4-7026-4bdc-a2ff-05a6a0c5378e None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:40.285216 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 173/347] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:11:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:40.291449 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548602f2-102d-48dc-8b24-47910e1df108 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:40.291726 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548602f2-102d-48dc-8b24-47910e1df108 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:40.291996 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548602f2-102d-48dc-8b24-47910e1df108 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:40.293536 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 174/348] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:40 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:40.294242 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-548602f2-102d-48dc-8b24-47910e1df108 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:40.312816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54589c50-adbe-4e88-89b4-d706caa5d2d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:40.313575 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54589c50-adbe-4e88-89b4-d706caa5d2d1 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:40.313854 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54589c50-adbe-4e88-89b4-d706caa5d2d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:40.314222 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54589c50-adbe-4e88-89b4-d706caa5d2d1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:40.328678 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54589c50-adbe-4e88-89b4-d706caa5d2d1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:40.330170 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54589c50-adbe-4e88-89b4-d706caa5d2d1 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:40.347534 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-54589c50-adbe-4e88-89b4-d706caa5d2d1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:40.347534 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:40.347534 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:40.347534 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:40.347534 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:40.347534 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:40.347534 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:40.347534 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:40.347534 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:40.347534 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:40.347534 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:40.347534 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:40.347534 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:11:40.347534 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:11:40.347534 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:11:40.347534 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:11:40.349104 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:40.349104 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:40.349104 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:40.349104 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:40.349104 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:40.349104 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:40.349104 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:40.349104 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:40.349104 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:40.349104 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:40.349104 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:40.349104 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:40.349104 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:40.349104 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:40.349104 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:40.349104 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:40.349104 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:11:40.350722 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:11:40.350722 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:40.350722 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:40.350722 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 175/349] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:11:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 281 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:40.350722 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c55c59-7071-42a1-8502-9fa1fa2afdd7 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:40.351006 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c55c59-7071-42a1-8502-9fa1fa2afdd7 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:40.351241 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c55c59-7071-42a1-8502-9fa1fa2afdd7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:40.372252 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-54c55c59-7071-42a1-8502-9fa1fa2afdd7 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:40.374024 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-54c55c59-7071-42a1-8502-9fa1fa2afdd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:40.406925 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:40.437517 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-548602f2-102d-48dc-8b24-47910e1df108 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:40.464774 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 176/350] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:11:40.464774 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c55c59-7071-42a1-8502-9fa1fa2afdd7 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:40.490476 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c63b0d-6dad-4f3b-8821-6605727bca00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:40.491088 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c63b0d-6dad-4f3b-8821-6605727bca00 None admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:40.491422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c63b0d-6dad-4f3b-8821-6605727bca00 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:40.491694 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c63b0d-6dad-4f3b-8821-6605727bca00 None admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:40.492346 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c63b0d-6dad-4f3b-8821-6605727bca00 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:40.495296 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c63b0d-6dad-4f3b-8821-6605727bca00 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:40.533480 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 175/351] 10.222.0.42 () {60 vars in 1132 bytes} [Thu Oct 27 01:11:40 2022] POST /identity/v3/users => generated 248 bytes in 239 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:11:40.623786 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c55c59-7071-42a1-8502-9fa1fa2afdd7 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:40.669168 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98be33a-c171-4041-ac6b-afe4d3500a04 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:40.669496 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98be33a-c171-4041-ac6b-afe4d3500a04 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:40.669759 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98be33a-c171-4041-ac6b-afe4d3500a04 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:40.671203 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 176/352] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 27 01:11:40.672533 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df3ea1a-abdf-4641-b74b-b8423f2a00ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:40.673189 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df3ea1a-abdf-4641-b74b-b8423f2a00ad None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:40.673462 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df3ea1a-abdf-4641-b74b-b8423f2a00ad None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:40.673751 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df3ea1a-abdf-4641-b74b-b8423f2a00ad None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:40.674412 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df3ea1a-abdf-4641-b74b-b8423f2a00ad None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:40.674709 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df3ea1a-abdf-4641-b74b-b8423f2a00ad None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:40.676155 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df3ea1a-abdf-4641-b74b-b8423f2a00ad None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:40.682881 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a03d7f4-3fa0-45f2-917d-3dc16724e5aa None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:40.683159 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a03d7f4-3fa0-45f2-917d-3dc16724e5aa None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:40.683430 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a03d7f4-3fa0-45f2-917d-3dc16724e5aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:40.705339 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 177/353] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:11:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:40.710158 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-5a03d7f4-3fa0-45f2-917d-3dc16724e5aa None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:40.711053 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df3ea1a-abdf-4641-b74b-b8423f2a00ad None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:40.715095 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-5a03d7f4-3fa0-45f2-917d-3dc16724e5aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:40.745747 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:40.762446 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:40.842576 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 177/354] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 27 01:11:40.843323 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a03d7f4-3fa0-45f2-917d-3dc16724e5aa None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:40.869552 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df3ea1a-abdf-4641-b74b-b8423f2a00ad None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:40.917754 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806670ab-302b-4e9f-afa7-0138f75903ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:40.918368 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806670ab-302b-4e9f-afa7-0138f75903ab None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:40.918651 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806670ab-302b-4e9f-afa7-0138f75903ab None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:40.918920 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806670ab-302b-4e9f-afa7-0138f75903ab None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:40.919566 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-806670ab-302b-4e9f-afa7-0138f75903ab None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:40.921013 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-806670ab-302b-4e9f-afa7-0138f75903ab None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:40.971956 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 178/355] 10.222.0.42 () {60 vars in 1132 bytes} [Thu Oct 27 01:11:40 2022] POST /identity/v3/users => generated 248 bytes in 260 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:11:41.086626 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a03d7f4-3fa0-45f2-917d-3dc16724e5aa None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:41.151791 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39670eb5-fa21-4d6b-8c0e-89d08e2a3723 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:41.152438 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39670eb5-fa21-4d6b-8c0e-89d08e2a3723 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:41.152713 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39670eb5-fa21-4d6b-8c0e-89d08e2a3723 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:41.152977 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39670eb5-fa21-4d6b-8c0e-89d08e2a3723 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:41.168729 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39670eb5-fa21-4d6b-8c0e-89d08e2a3723 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:41.170148 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39670eb5-fa21-4d6b-8c0e-89d08e2a3723 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:41.189436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-39670eb5-fa21-4d6b-8c0e-89d08e2a3723 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:41.189436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:41.189436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:41.189436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:41.189436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:41.189436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:41.189436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:41.189436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:41.189436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:41.189436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:41.189436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:41.189436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:41.189436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:11:41.189436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:11:41.189436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:11:41.189436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:11:41.190395 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:41.190395 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:41.190395 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:41.190395 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:41.190395 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:41.190395 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:41.190395 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:41.190395 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:41.190395 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:41.190395 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:41.190395 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:41.190395 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:41.190395 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:41.190395 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:41.190395 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:41.190395 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:41.190395 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:11:41.191231 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:11:41.191231 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:41.191231 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:41.191231 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 178/356] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:11:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 348 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:41.197019 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806670ab-302b-4e9f-afa7-0138f75903ab None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:41.200405 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39670eb5-fa21-4d6b-8c0e-89d08e2a3723 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:41.213771 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:41.323259 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:41.335358 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39670eb5-fa21-4d6b-8c0e-89d08e2a3723 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:41.369952 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806670ab-302b-4e9f-afa7-0138f75903ab None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:41.397383 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7c43f6-bbec-405e-aea5-9ecc7f76e688 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:41.400773 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7c43f6-bbec-405e-aea5-9ecc7f76e688 None admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:41.401270 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7c43f6-bbec-405e-aea5-9ecc7f76e688 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:41.401760 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7c43f6-bbec-405e-aea5-9ecc7f76e688 None admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:41.402668 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7c43f6-bbec-405e-aea5-9ecc7f76e688 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:41.405667 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7c43f6-bbec-405e-aea5-9ecc7f76e688 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:41.417608 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.sql.core [None req-8a7c43f6-bbec-405e-aea5-9ecc7f76e688 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 27 01:11:41.417608 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 27 01:11:41.417608 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [parameters: {'id': '6de43a74d15f485e912b3ad907acffa6', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 27 01:11:41.417608 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81588) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 27 01:11:41.425036 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-8a7c43f6-bbec-405e-aea5-9ecc7f76e688 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 27 01:11:41.425036 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:41.425036 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 27 01:11:41.425036 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 27 01:11:41.425036 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 27 01:11:41.425036 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 27 01:11:41.425036 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 27 01:11:41.425036 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application result = self._query(query) Oct 27 01:11:41.425036 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 27 01:11:41.425036 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application conn.query(q) Oct 27 01:11:41.425036 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 27 01:11:41.425036 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 27 01:11:41.425036 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 27 01:11:41.425036 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application result.read() Oct 27 01:11:41.425036 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 27 01:11:41.426506 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 27 01:11:41.426506 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 27 01:11:41.426506 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 27 01:11:41.426506 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 27 01:11:41.426506 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 27 01:11:41.426506 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 27 01:11:41.426506 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 27 01:11:41.426506 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 27 01:11:41.426506 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:41.426506 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 27 01:11:41.426506 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:41.426506 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:41.426506 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 27 01:11:41.426506 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 27 01:11:41.426506 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Oct 27 01:11:41.426506 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return ref.to_dict() Oct 27 01:11:41.426506 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 27 01:11:41.426506 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application next(self.gen) Oct 27 01:11:41.427420 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1068, in _transaction_scope Oct 27 01:11:41.427420 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application yield resource Oct 27 01:11:41.427420 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 27 01:11:41.427420 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application next(self.gen) Oct 27 01:11:41.427420 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 670, in _session Oct 27 01:11:41.427420 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application self.session.rollback() Oct 27 01:11:41.427420 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 27 01:11:41.427420 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application self.force_reraise() Oct 27 01:11:41.427420 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 27 01:11:41.427420 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise self.value Oct 27 01:11:41.427420 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 667, in _session Oct 27 01:11:41.427420 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 27 01:11:41.427420 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 695, in _end_session_transaction Oct 27 01:11:41.427420 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application session.commit() Oct 27 01:11:41.427420 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 27 01:11:41.427420 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 27 01:11:41.427420 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 27 01:11:41.428250 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application self._prepare_impl() Oct 27 01:11:41.428250 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 27 01:11:41.428250 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application self.session.flush() Oct 27 01:11:41.428250 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 27 01:11:41.428250 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application self._flush(objects) Oct 27 01:11:41.428250 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 27 01:11:41.428250 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 27 01:11:41.428250 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 27 01:11:41.428250 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application compat.raise_( Oct 27 01:11:41.428250 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 27 01:11:41.428250 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception Oct 27 01:11:41.428250 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 27 01:11:41.428250 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application flush_context.execute() Oct 27 01:11:41.428250 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 27 01:11:41.428250 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rec.execute(self) Oct 27 01:11:41.428250 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 27 01:11:41.428250 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 27 01:11:41.429148 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 27 01:11:41.429148 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 27 01:11:41.429148 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 27 01:11:41.429148 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 27 01:11:41.429148 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 27 01:11:41.429148 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 27 01:11:41.429148 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 27 01:11:41.429148 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 27 01:11:41.429148 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 27 01:11:41.429148 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 27 01:11:41.429148 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 27 01:11:41.429148 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 27 01:11:41.429148 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 27 01:11:41.429148 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 27 01:11:41.429148 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 27 01:11:41.429148 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception Oct 27 01:11:41.430029 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 27 01:11:41.430029 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 27 01:11:41.430029 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 27 01:11:41.430029 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 27 01:11:41.430029 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 27 01:11:41.430029 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application result = self._query(query) Oct 27 01:11:41.430029 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 27 01:11:41.430029 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application conn.query(q) Oct 27 01:11:41.430029 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 27 01:11:41.430029 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 27 01:11:41.430029 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 27 01:11:41.430029 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application result.read() Oct 27 01:11:41.430029 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 27 01:11:41.430029 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 27 01:11:41.430029 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 27 01:11:41.430029 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 27 01:11:41.430794 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 27 01:11:41.430794 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 27 01:11:41.430794 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 27 01:11:41.430794 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 27 01:11:41.430794 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 27 01:11:41.430794 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 27 01:11:41.430794 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [parameters: {'id': '6de43a74d15f485e912b3ad907acffa6', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 27 01:11:41.430794 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 27 01:11:41.430794 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:41.430794 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 27 01:11:41.430794 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:41.430794 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:41.430794 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:41.430794 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:41.430794 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:41.430794 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:41.431862 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:41.431862 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:41.431862 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:41.431862 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:41.431862 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:41.431862 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:41.431862 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Oct 27 01:11:41.431862 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Oct 27 01:11:41.431862 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:41.431862 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:41.431862 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Oct 27 01:11:41.431862 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Oct 27 01:11:41.431862 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 27 01:11:41.431862 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 27 01:11:41.431862 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 27 01:11:41.431862 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:41.441889 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 179/357] 10.222.0.42 () {60 vars in 1131 bytes} [Thu Oct 27 01:11:41 2022] POST /identity/v3/roles => generated 144 bytes in 241 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Oct 27 01:11:41.442937 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6e8f32-9c5d-4ada-8973-ed61e35d3423 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:41.443833 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6e8f32-9c5d-4ada-8973-ed61e35d3423 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:41.444325 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6e8f32-9c5d-4ada-8973-ed61e35d3423 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:41.444828 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6e8f32-9c5d-4ada-8973-ed61e35d3423 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:41.446025 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e6e8f32-9c5d-4ada-8973-ed61e35d3423 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:41.447079 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7c43f6-bbec-405e-aea5-9ecc7f76e688 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:41.447584 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e6e8f32-9c5d-4ada-8973-ed61e35d3423 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:41.449181 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e6e8f32-9c5d-4ada-8973-ed61e35d3423 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:41.480046 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 179/358] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:11:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:41.484424 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6e8f32-9c5d-4ada-8973-ed61e35d3423 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:41.545241 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:41.617325 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6e8f32-9c5d-4ada-8973-ed61e35d3423 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:41.618465 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7c43f6-bbec-405e-aea5-9ecc7f76e688 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:41.653142 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606ea0a9-57b0-494d-82e1-1f5789706d82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:41.657083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606ea0a9-57b0-494d-82e1-1f5789706d82 None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:41.657083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606ea0a9-57b0-494d-82e1-1f5789706d82 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:41.657083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606ea0a9-57b0-494d-82e1-1f5789706d82 None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:41.657083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606ea0a9-57b0-494d-82e1-1f5789706d82 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:41.657083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606ea0a9-57b0-494d-82e1-1f5789706d82 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:41.667145 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:41.682689 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2383d0ff-633d-4466-8882-403650edc535 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:41.683341 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2383d0ff-633d-4466-8882-403650edc535 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:41.683587 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2383d0ff-633d-4466-8882-403650edc535 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:41.683854 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2383d0ff-633d-4466-8882-403650edc535 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:41.691305 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 180/359] 10.222.0.42 () {60 vars in 1132 bytes} [Thu Oct 27 01:11:41 2022] POST /identity/v3/users => generated 249 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:11:41.704807 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2383d0ff-633d-4466-8882-403650edc535 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:41.711303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-2383d0ff-633d-4466-8882-403650edc535 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 27 01:11:41.711303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:41.711303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:41.711303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:41.711303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:41.711303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:41.711303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:41.711303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:41.711303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:41.711303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:41.711303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:41.711303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:41.711303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:41.711303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:41.711303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:41.711303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:41.712270 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:41.712270 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:41.712270 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:41.712270 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:41.712270 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:41.712270 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:41.712270 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:41.712270 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:41.712270 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:41.712270 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:41.712270 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:41.712270 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:41.712270 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:41.712270 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:41.712270 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:41.712270 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:41.713086 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:41.713086 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:41.713086 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:41.713086 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:41.713086 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:41.713086 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:41.713086 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:41.713086 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:41.713086 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:41.713086 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:41.713086 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 27 01:11:41.713086 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:41.713086 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 180/360] 10.222.0.42 () {56 vars in 1114 bytes} [Thu Oct 27 01:11:41 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 266 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:41.718729 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606ea0a9-57b0-494d-82e1-1f5789706d82 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:41.881902 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606ea0a9-57b0-494d-82e1-1f5789706d82 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:41.925033 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f078d96f-8a4f-490a-843f-d27f80833454 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:41.925680 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f078d96f-8a4f-490a-843f-d27f80833454 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:41.926474 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f078d96f-8a4f-490a-843f-d27f80833454 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:41.927485 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f078d96f-8a4f-490a-843f-d27f80833454 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:41.928179 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f078d96f-8a4f-490a-843f-d27f80833454 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:41.928260 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f078d96f-8a4f-490a-843f-d27f80833454 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:41.929622 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f078d96f-8a4f-490a-843f-d27f80833454 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:41.938592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 181/361] 10.222.0.42 () {56 vars in 1111 bytes} [Thu Oct 27 01:11:41 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:42.730186 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cbff5ca-5d57-4aaa-8923-b824b3da48f5 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:42.730728 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cbff5ca-5d57-4aaa-8923-b824b3da48f5 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:42.730728 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cbff5ca-5d57-4aaa-8923-b824b3da48f5 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:42.732294 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 181/362] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:42.737790 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7ce3fe-51e8-4e14-b9d5-050b4080f993 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:42.738201 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7ce3fe-51e8-4e14-b9d5-050b4080f993 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:42.738433 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7ce3fe-51e8-4e14-b9d5-050b4080f993 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:42.769192 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-5a7ce3fe-51e8-4e14-b9d5-050b4080f993 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:42.770469 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-5a7ce3fe-51e8-4e14-b9d5-050b4080f993 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:42.851570 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:42.875463 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 182/363] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:11:43.249541 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cbff5ca-5d57-4aaa-8923-b824b3da48f5 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:43.332628 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574f6907-c195-45e2-9ade-02cf31e4efa3 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:43.333915 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574f6907-c195-45e2-9ade-02cf31e4efa3 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:43.334406 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574f6907-c195-45e2-9ade-02cf31e4efa3 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:43.335932 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 183/364] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:43 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:43.345096 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6484c916-9d14-4581-9113-19eb5cbe8b26 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:43.345096 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6484c916-9d14-4581-9113-19eb5cbe8b26 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:43.345096 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6484c916-9d14-4581-9113-19eb5cbe8b26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:43.373025 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-6484c916-9d14-4581-9113-19eb5cbe8b26 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:43.377575 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-6484c916-9d14-4581-9113-19eb5cbe8b26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:43.411897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cbff5ca-5d57-4aaa-8923-b824b3da48f5 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:43.484601 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:43.506051 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8132ba38-3fa5-4ad5-bdd2-e97f2f65a238 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:43.506934 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8132ba38-3fa5-4ad5-bdd2-e97f2f65a238 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:43.507360 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8132ba38-3fa5-4ad5-bdd2-e97f2f65a238 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:43.507776 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8132ba38-3fa5-4ad5-bdd2-e97f2f65a238 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:43.533584 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8132ba38-3fa5-4ad5-bdd2-e97f2f65a238 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:43.539180 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-8132ba38-3fa5-4ad5-bdd2-e97f2f65a238 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 27 01:11:43.539180 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:43.539180 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:43.539180 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:43.539180 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:43.539180 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:43.539180 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:43.539180 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:43.539180 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:43.539180 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:43.539180 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:43.539180 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:43.539180 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:43.539180 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:43.539180 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:43.539180 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:43.540150 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:43.540150 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:43.540150 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:43.540150 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:43.540150 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:43.540150 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:43.540150 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:43.540150 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:43.540150 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:43.540150 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:43.540150 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:43.540150 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:43.540150 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:43.540150 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:43.540150 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:43.540150 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:43.540983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:43.540983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:43.540983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:43.540983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:43.540983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:43.540983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:43.540983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:43.540983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:43.540983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:43.540983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:43.540983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 27 01:11:43.540983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:43.552535 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 184/365] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:11:43.553031 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 182/366] 10.222.0.42 () {56 vars in 1102 bytes} [Thu Oct 27 01:11:43 2022] GET /identity/v3/roles/service => generated 85 bytes in 304 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:43.558283 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6484c916-9d14-4581-9113-19eb5cbe8b26 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:43.710852 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bd7dda-51a1-4279-a9e8-462dec7dbc8d None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:43.711400 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bd7dda-51a1-4279-a9e8-462dec7dbc8d None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:43.711831 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bd7dda-51a1-4279-a9e8-462dec7dbc8d None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:43.713353 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 183/367] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 27 01:11:43.717513 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6484c916-9d14-4581-9113-19eb5cbe8b26 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:43.726006 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776731c7-35a1-4595-aab4-9b744c4d27d6 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:43.726505 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776731c7-35a1-4595-aab4-9b744c4d27d6 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:43.726942 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776731c7-35a1-4595-aab4-9b744c4d27d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:43.763444 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-776731c7-35a1-4595-aab4-9b744c4d27d6 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:43.768349 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-776731c7-35a1-4595-aab4-9b744c4d27d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:43.776662 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:43.838673 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9760f8-15fc-4191-9e08-1fd768093d0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:43.839300 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9760f8-15fc-4191-9e08-1fd768093d0e None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:43.839577 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9760f8-15fc-4191-9e08-1fd768093d0e None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:43.839849 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9760f8-15fc-4191-9e08-1fd768093d0e None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:43.840524 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d9760f8-15fc-4191-9e08-1fd768093d0e None admin] RBAC: Adding query filter params (name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:43.840823 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d9760f8-15fc-4191-9e08-1fd768093d0e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:43.842248 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d9760f8-15fc-4191-9e08-1fd768093d0e None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:43.871394 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 185/368] 10.222.0.42 () {56 vars in 1103 bytes} [Thu Oct 27 01:11:43 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 313 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:43.876514 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9760f8-15fc-4191-9e08-1fd768093d0e None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:43.972571 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 184/369] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 247 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 27 01:11:43.975168 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-776731c7-35a1-4595-aab4-9b744c4d27d6 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:44.106753 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9760f8-15fc-4191-9e08-1fd768093d0e None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:44.177605 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146ff119-10f1-496d-8ac6-3e90fccef6f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:44.179467 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146ff119-10f1-496d-8ac6-3e90fccef6f0 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:44.181914 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-776731c7-35a1-4595-aab4-9b744c4d27d6 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:44.182302 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146ff119-10f1-496d-8ac6-3e90fccef6f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:44.182679 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146ff119-10f1-496d-8ac6-3e90fccef6f0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:44.201863 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-146ff119-10f1-496d-8ac6-3e90fccef6f0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:44.203321 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-146ff119-10f1-496d-8ac6-3e90fccef6f0 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:44.222048 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:44.227293 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-146ff119-10f1-496d-8ac6-3e90fccef6f0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:44.227293 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:44.227293 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:44.227293 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:44.227293 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:44.227293 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:44.227293 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:44.227293 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:44.227293 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:44.227293 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:44.227293 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:44.227293 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:44.227293 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:11:44.227293 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:11:44.227293 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:11:44.227293 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:11:44.228463 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:44.228463 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:44.228463 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:44.228463 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:44.228463 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:44.228463 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:44.228463 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:44.228463 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:44.228463 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:44.228463 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:44.228463 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:44.228463 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:44.228463 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:44.228463 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:44.228463 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:44.228463 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:44.228463 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:11:44.229438 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:11:44.229438 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:44.229438 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:44.239188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 186/370] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:11:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 362 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 27 01:11:44.240089 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146ff119-10f1-496d-8ac6-3e90fccef6f0 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:44.260013 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ed9e84-1cc9-4996-9464-0874b6d49948 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:44.260646 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ed9e84-1cc9-4996-9464-0874b6d49948 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:44.260916 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ed9e84-1cc9-4996-9464-0874b6d49948 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:44.261218 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ed9e84-1cc9-4996-9464-0874b6d49948 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:44.280477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ed9e84-1cc9-4996-9464-0874b6d49948 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:44.285723 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-10ed9e84-1cc9-4996-9464-0874b6d49948 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 27 01:11:44.285723 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:44.285723 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:44.285723 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:44.285723 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:44.285723 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:44.285723 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:44.285723 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:44.285723 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:44.285723 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:44.285723 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:44.285723 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:44.285723 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:44.285723 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:44.285723 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:44.285723 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:44.287418 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:44.287418 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:44.287418 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:44.287418 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:44.287418 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:44.287418 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:44.287418 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:44.287418 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:44.287418 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:44.287418 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:44.287418 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:44.287418 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:44.287418 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:44.287418 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:44.287418 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:44.287418 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:44.289146 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:44.289146 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:44.289146 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:44.289146 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:44.289146 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:44.289146 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:44.289146 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:44.289146 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:44.289146 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:44.289146 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:44.289146 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 27 01:11:44.289146 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:44.289146 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 185/371] 10.222.0.42 () {56 vars in 1102 bytes} [Thu Oct 27 01:11:43 2022] GET /identity/v3/roles/service => generated 85 bytes in 315 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:44.293020 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ed9e84-1cc9-4996-9464-0874b6d49948 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:44.301640 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:44.389918 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146ff119-10f1-496d-8ac6-3e90fccef6f0 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:44.420697 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ed9e84-1cc9-4996-9464-0874b6d49948 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:44.436195 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14197d56-1596-412e-ae03-2c30e0be925a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:44.437093 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14197d56-1596-412e-ae03-2c30e0be925a None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:44.437377 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14197d56-1596-412e-ae03-2c30e0be925a None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:44.437647 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14197d56-1596-412e-ae03-2c30e0be925a None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:44.438294 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14197d56-1596-412e-ae03-2c30e0be925a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:44.438859 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14197d56-1596-412e-ae03-2c30e0be925a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:44.440968 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14197d56-1596-412e-ae03-2c30e0be925a None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:44.456338 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 187/372] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:11:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:44.457213 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14197d56-1596-412e-ae03-2c30e0be925a None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:44.466694 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-876db3dd-1a5f-468b-ae2b-b9e8ef47f8ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:44.467294 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876db3dd-1a5f-468b-ae2b-b9e8ef47f8ef None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:44.467573 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876db3dd-1a5f-468b-ae2b-b9e8ef47f8ef None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:44.467841 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876db3dd-1a5f-468b-ae2b-b9e8ef47f8ef None admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:44.468500 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-876db3dd-1a5f-468b-ae2b-b9e8ef47f8ef None admin] RBAC: Adding query filter params (name=service) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:44.468794 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-876db3dd-1a5f-468b-ae2b-b9e8ef47f8ef None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:44.470213 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-876db3dd-1a5f-468b-ae2b-b9e8ef47f8ef None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:44.483314 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 186/373] 10.222.0.42 () {56 vars in 1103 bytes} [Thu Oct 27 01:11:44 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:44.483314 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f26a9b-4724-499e-9116-eb2c506eccbc None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:44.483314 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f26a9b-4724-499e-9116-eb2c506eccbc None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:44.483314 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f26a9b-4724-499e-9116-eb2c506eccbc None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:44.483314 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 187/374] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:44.484206 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10f26a9b-4724-499e-9116-eb2c506eccbc None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:44.526221 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:44.555639 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:44.566102 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14197d56-1596-412e-ae03-2c30e0be925a None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:44.590842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10f26a9b-4724-499e-9116-eb2c506eccbc None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:44.601926 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b03335bb-ec11-4298-9fbe-a2e44e26e4ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:44.602559 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03335bb-ec11-4298-9fbe-a2e44e26e4ac None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:44.602835 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03335bb-ec11-4298-9fbe-a2e44e26e4ac None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:44.603126 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03335bb-ec11-4298-9fbe-a2e44e26e4ac None admin] PATH_INFO: `/v3/roles/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:44.612958 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b03335bb-ec11-4298-9fbe-a2e44e26e4ac None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:44.618418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-b03335bb-ec11-4298-9fbe-a2e44e26e4ac None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 27 01:11:44.618418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:44.618418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:44.618418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:44.618418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:44.618418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:44.618418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:44.618418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:44.618418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:44.618418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:44.618418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:44.618418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:44.618418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:44.618418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:44.618418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:44.618418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:44.619307 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:44.619307 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:44.619307 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:44.619307 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:44.619307 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:44.619307 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:44.619307 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:44.619307 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:44.619307 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:44.619307 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:44.619307 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:44.619307 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:44.619307 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:44.619307 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:44.619307 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:44.619307 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:44.620098 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:44.620098 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:44.620098 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:44.620098 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:44.620098 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:44.620098 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:44.620098 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:44.620098 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:44.620098 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:44.620098 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:44.620098 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 27 01:11:44.620098 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:44.620098 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 188/375] 10.222.0.42 () {56 vars in 1102 bytes} [Thu Oct 27 01:11:44 2022] GET /identity/v3/roles/service => generated 85 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:44.620766 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b03335bb-ec11-4298-9fbe-a2e44e26e4ac None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:44.631739 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d33a16-fa71-42d9-a15a-9c2994ce394c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:44.632368 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d33a16-fa71-42d9-a15a-9c2994ce394c None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:44.632649 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d33a16-fa71-42d9-a15a-9c2994ce394c None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:44.632920 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d33a16-fa71-42d9-a15a-9c2994ce394c None admin] PATH_INFO: `/v3/users/nova` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:44.640429 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d33a16-fa71-42d9-a15a-9c2994ce394c None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:44.641840 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d33a16-fa71-42d9-a15a-9c2994ce394c None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:44.654531 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-53d33a16-fa71-42d9-a15a-9c2994ce394c None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 27 01:11:44.654531 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:44.654531 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:44.654531 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:44.654531 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:44.654531 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:44.654531 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:44.654531 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:44.654531 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:44.654531 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:44.654531 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:44.654531 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:44.654531 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 27 01:11:44.654531 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 27 01:11:44.654531 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 27 01:11:44.654531 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 27 01:11:44.655439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:44.655439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:44.655439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 27 01:11:44.655439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:11:44.655439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 27 01:11:44.655439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:11:44.655439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:44.655439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:44.655439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:44.655439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:44.655439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:44.655439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:44.655439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:44.655439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:44.655439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:44.655439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:44.655439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:44.655439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:44.656338 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:44.656338 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:44.656338 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 27 01:11:44.656338 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 27 01:11:44.656338 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 27 01:11:44.656338 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 27 01:11:44.656338 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 27 01:11:44.656338 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 27 01:11:44.656338 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 27 01:11:44.656338 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:44.656338 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 188/376] 10.222.0.42 () {56 vars in 1093 bytes} [Thu Oct 27 01:11:44 2022] GET /identity/v3/users/nova => generated 82 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:44.657447 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94db47f4-272c-419d-a6c7-252f75fcf450 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:44.657447 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94db47f4-272c-419d-a6c7-252f75fcf450 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:44.657683 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94db47f4-272c-419d-a6c7-252f75fcf450 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:44.678429 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-94db47f4-272c-419d-a6c7-252f75fcf450 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:44.679684 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-94db47f4-272c-419d-a6c7-252f75fcf450 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:44.734524 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b03335bb-ec11-4298-9fbe-a2e44e26e4ac None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:44.757615 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 189/377] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:11:44.757615 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94db47f4-272c-419d-a6c7-252f75fcf450 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:44.772351 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef4b26e-7204-4b35-93cf-aed82dbaa5f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:44.772982 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef4b26e-7204-4b35-93cf-aed82dbaa5f8 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:44.773280 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef4b26e-7204-4b35-93cf-aed82dbaa5f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:44.773546 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef4b26e-7204-4b35-93cf-aed82dbaa5f8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:44.785710 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ef4b26e-7204-4b35-93cf-aed82dbaa5f8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:44.787331 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ef4b26e-7204-4b35-93cf-aed82dbaa5f8 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:44.795232 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:44.804816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-9ef4b26e-7204-4b35-93cf-aed82dbaa5f8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:44.804816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:44.804816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:44.804816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:44.804816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:44.804816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:44.804816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:44.804816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:44.804816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:44.804816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:44.804816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:44.804816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:44.804816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:11:44.804816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:11:44.804816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:11:44.804816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:11:44.805827 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:44.805827 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:44.805827 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:44.805827 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:44.805827 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:44.805827 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:44.805827 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:44.805827 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:44.805827 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:44.805827 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:44.805827 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:44.805827 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:44.805827 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:44.805827 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:44.805827 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:44.805827 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:44.805827 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:11:44.806912 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:11:44.806912 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:44.806912 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:44.806912 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 189/378] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:11:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:44.807536 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef4b26e-7204-4b35-93cf-aed82dbaa5f8 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:44.810458 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:44.886243 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94db47f4-272c-419d-a6c7-252f75fcf450 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:44.913878 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef4b26e-7204-4b35-93cf-aed82dbaa5f8 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:44.922177 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b43d3e5-4176-4bbd-a5c0-39acc9480b15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:44.922781 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b43d3e5-4176-4bbd-a5c0-39acc9480b15 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:44.923057 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b43d3e5-4176-4bbd-a5c0-39acc9480b15 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:44.923326 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b43d3e5-4176-4bbd-a5c0-39acc9480b15 None admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:44.923986 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b43d3e5-4176-4bbd-a5c0-39acc9480b15 None admin] RBAC: Adding query filter params (name=service) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:44.924285 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b43d3e5-4176-4bbd-a5c0-39acc9480b15 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:44.925707 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b43d3e5-4176-4bbd-a5c0-39acc9480b15 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:44.934683 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 190/379] 10.222.0.42 () {56 vars in 1103 bytes} [Thu Oct 27 01:11:44 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:44.935729 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acae75c7-ed89-4095-b81a-8e22b87f2765 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:44.936015 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acae75c7-ed89-4095-b81a-8e22b87f2765 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:44.936288 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acae75c7-ed89-4095-b81a-8e22b87f2765 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:44.937622 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 191/380] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:44 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:44.938178 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acae75c7-ed89-4095-b81a-8e22b87f2765 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:44.955431 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d2c243-efd3-4df3-b6a2-19915b26ec3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:44.956036 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d2c243-efd3-4df3-b6a2-19915b26ec3b None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:44.956307 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d2c243-efd3-4df3-b6a2-19915b26ec3b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:44.956571 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d2c243-efd3-4df3-b6a2-19915b26ec3b None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:44.958330 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d2c243-efd3-4df3-b6a2-19915b26ec3b None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:44.958330 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d2c243-efd3-4df3-b6a2-19915b26ec3b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:44.959569 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d2c243-efd3-4df3-b6a2-19915b26ec3b None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:44.996907 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 190/381] 10.222.0.42 () {56 vars in 1133 bytes} [Thu Oct 27 01:11:44 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:44.997608 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d2c243-efd3-4df3-b6a2-19915b26ec3b None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:45.039082 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:45.050292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acae75c7-ed89-4095-b81a-8e22b87f2765 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:45.061694 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:45.084384 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d18309-6c17-465b-8bbf-34a38c4a3ab4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:45.085005 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d18309-6c17-465b-8bbf-34a38c4a3ab4 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:45.085312 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d18309-6c17-465b-8bbf-34a38c4a3ab4 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:45.085579 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d18309-6c17-465b-8bbf-34a38c4a3ab4 None admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:45.086208 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d18309-6c17-465b-8bbf-34a38c4a3ab4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:45.086504 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d18309-6c17-465b-8bbf-34a38c4a3ab4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:45.087849 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d18309-6c17-465b-8bbf-34a38c4a3ab4 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:45.099411 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 192/382] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:11:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:45.100401 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74557666-8b51-43ed-9da3-b9d76df813d0 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:45.100681 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74557666-8b51-43ed-9da3-b9d76df813d0 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:45.100949 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74557666-8b51-43ed-9da3-b9d76df813d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:45.106476 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d2c243-efd3-4df3-b6a2-19915b26ec3b None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:45.118191 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-74557666-8b51-43ed-9da3-b9d76df813d0 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:45.119380 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-74557666-8b51-43ed-9da3-b9d76df813d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:45.140152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1949979f-8606-4196-b172-b691a737fb91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:45.140838 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1949979f-8606-4196-b172-b691a737fb91 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:45.141151 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1949979f-8606-4196-b172-b691a737fb91 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:45.141454 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1949979f-8606-4196-b172-b691a737fb91 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:45.151186 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1949979f-8606-4196-b172-b691a737fb91 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:45.152660 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1949979f-8606-4196-b172-b691a737fb91 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:45.167352 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-1949979f-8606-4196-b172-b691a737fb91 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:45.167352 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:45.167352 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:45.167352 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:45.167352 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:45.167352 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:45.167352 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:45.167352 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:45.167352 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:45.167352 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:45.167352 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:45.167352 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:45.167352 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:11:45.167352 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:11:45.167352 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:11:45.167352 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:11:45.168227 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:45.168227 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:45.168227 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:45.168227 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:45.168227 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:45.168227 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:45.168227 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:45.168227 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:45.168227 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:45.168227 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:45.168227 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:45.168227 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:45.168227 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:45.168227 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:45.168227 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:45.168227 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:45.168227 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:11:45.169101 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:11:45.169101 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:45.169101 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:45.169101 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 191/383] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:11:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:45.169319 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1949979f-8606-4196-b172-b691a737fb91 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:45.188888 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 193/384] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:11:45.189750 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74557666-8b51-43ed-9da3-b9d76df813d0 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:45.272061 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:45.291048 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1949979f-8606-4196-b172-b691a737fb91 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:45.292135 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74557666-8b51-43ed-9da3-b9d76df813d0 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:45.306841 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:45.328882 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6c6e22-0c99-4ea4-902f-9faa7bf42e13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:45.329542 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6c6e22-0c99-4ea4-902f-9faa7bf42e13 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:45.329820 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6c6e22-0c99-4ea4-902f-9faa7bf42e13 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:45.330097 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6c6e22-0c99-4ea4-902f-9faa7bf42e13 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:45.331182 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbddafc0-c30e-40d4-bd3f-5c230386b9f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:45.332468 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbddafc0-c30e-40d4-bd3f-5c230386b9f2 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:45.332768 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbddafc0-c30e-40d4-bd3f-5c230386b9f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:45.333038 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbddafc0-c30e-40d4-bd3f-5c230386b9f2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:45.341374 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6c6e22-0c99-4ea4-902f-9faa7bf42e13 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:45.344080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbddafc0-c30e-40d4-bd3f-5c230386b9f2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:45.345619 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbddafc0-c30e-40d4-bd3f-5c230386b9f2 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:45.346598 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-cb6c6e22-0c99-4ea4-902f-9faa7bf42e13 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 27 01:11:45.346598 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:45.346598 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:45.346598 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:45.346598 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:45.346598 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:45.346598 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:45.346598 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:45.346598 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:45.346598 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:45.346598 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:45.346598 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:45.346598 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:45.346598 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:45.346598 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:45.346598 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:45.347429 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:45.347429 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:45.347429 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:45.347429 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:45.347429 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:45.347429 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:45.347429 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:45.347429 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:45.347429 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:45.347429 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:45.347429 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:45.347429 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:45.347429 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:45.347429 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:45.347429 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:45.347429 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:45.348282 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:45.348282 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:45.348282 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:45.348282 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:45.348282 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:45.348282 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:45.348282 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:45.348282 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:45.348282 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:45.348282 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:45.348282 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 27 01:11:45.348282 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:45.348282 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 194/385] 10.222.0.42 () {56 vars in 1102 bytes} [Thu Oct 27 01:11:45 2022] GET /identity/v3/roles/service => generated 85 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:45.349708 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6c6e22-0c99-4ea4-902f-9faa7bf42e13 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:45.368984 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-cbddafc0-c30e-40d4-bd3f-5c230386b9f2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:45.368984 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:45.368984 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:45.368984 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:45.368984 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:45.368984 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:45.368984 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:45.368984 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:45.368984 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:45.368984 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:45.368984 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:45.368984 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:45.368984 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:11:45.368984 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:11:45.368984 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:11:45.368984 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:11:45.369998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:45.369998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:45.369998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:45.369998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:45.369998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:45.369998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:45.369998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:45.369998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:45.369998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:45.369998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:45.369998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:45.369998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:45.369998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:45.369998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:45.369998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:45.369998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:45.369998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:11:45.370935 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:11:45.370935 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:45.370935 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:45.370935 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 192/386] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:11:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:45.375357 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbddafc0-c30e-40d4-bd3f-5c230386b9f2 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:45.459684 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6c6e22-0c99-4ea4-902f-9faa7bf42e13 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:45.485580 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbddafc0-c30e-40d4-bd3f-5c230386b9f2 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:45.493394 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc203905-514b-4d7c-9ca1-32d9a8ef25b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:45.494010 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc203905-514b-4d7c-9ca1-32d9a8ef25b4 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:45.494283 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc203905-514b-4d7c-9ca1-32d9a8ef25b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:45.494541 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc203905-514b-4d7c-9ca1-32d9a8ef25b4 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:45.501303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc203905-514b-4d7c-9ca1-32d9a8ef25b4 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:45.502732 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc203905-514b-4d7c-9ca1-32d9a8ef25b4 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:45.505847 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:45.516389 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-bc203905-514b-4d7c-9ca1-32d9a8ef25b4 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 27 01:11:45.516389 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:45.516389 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:45.516389 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:45.516389 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:45.516389 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:45.516389 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:45.516389 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:45.516389 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:45.516389 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:45.516389 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:45.516389 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:45.516389 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 27 01:11:45.516389 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 27 01:11:45.516389 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 27 01:11:45.516389 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 27 01:11:45.517415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:45.517415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:45.517415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 27 01:11:45.517415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:11:45.517415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 27 01:11:45.517415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:11:45.517415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:45.517415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:45.517415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:45.517415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:45.517415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:45.517415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:45.517415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:45.517415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:45.517415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:45.517415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:45.517415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:45.517415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:45.518301 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:45.518301 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:45.518301 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 27 01:11:45.518301 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 27 01:11:45.518301 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 27 01:11:45.518301 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 27 01:11:45.518301 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 27 01:11:45.518301 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 27 01:11:45.518301 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 27 01:11:45.518301 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:45.518301 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 195/387] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:11:45 2022] GET /identity/v3/users/glance => generated 84 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:45.518855 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc203905-514b-4d7c-9ca1-32d9a8ef25b4 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:45.526382 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1436e4b-bfc9-4330-ba7a-7b0f62e378b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:45.527045 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1436e4b-bfc9-4330-ba7a-7b0f62e378b7 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:45.527319 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1436e4b-bfc9-4330-ba7a-7b0f62e378b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:45.527585 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1436e4b-bfc9-4330-ba7a-7b0f62e378b7 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:45.528261 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1436e4b-bfc9-4330-ba7a-7b0f62e378b7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:45.528560 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1436e4b-bfc9-4330-ba7a-7b0f62e378b7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:45.529951 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1436e4b-bfc9-4330-ba7a-7b0f62e378b7 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:45.543070 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 193/388] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:11:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:45.543860 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1436e4b-bfc9-4330-ba7a-7b0f62e378b7 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:45.568769 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:45.624202 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc203905-514b-4d7c-9ca1-32d9a8ef25b4 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:45.642598 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1436e4b-bfc9-4330-ba7a-7b0f62e378b7 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:45.661534 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94886724-771f-4d85-9d44-9158ea7da3ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:45.662138 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94886724-771f-4d85-9d44-9158ea7da3ce None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:45.662415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94886724-771f-4d85-9d44-9158ea7da3ce None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:45.662681 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94886724-771f-4d85-9d44-9158ea7da3ce None admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:45.663336 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94886724-771f-4d85-9d44-9158ea7da3ce None admin] RBAC: Adding query filter params (name=service) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:45.663630 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94886724-771f-4d85-9d44-9158ea7da3ce None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:45.664995 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94886724-771f-4d85-9d44-9158ea7da3ce None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:45.672763 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 196/389] 10.222.0.42 () {56 vars in 1103 bytes} [Thu Oct 27 01:11:45 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:45.677349 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94886724-771f-4d85-9d44-9158ea7da3ce None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:45.684380 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a410201-40c1-4ade-b5b5-34890b40ea29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:45.684983 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a410201-40c1-4ade-b5b5-34890b40ea29 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:45.685287 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a410201-40c1-4ade-b5b5-34890b40ea29 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:45.685556 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a410201-40c1-4ade-b5b5-34890b40ea29 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:45.686187 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a410201-40c1-4ade-b5b5-34890b40ea29 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:45.686485 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a410201-40c1-4ade-b5b5-34890b40ea29 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:45.687829 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a410201-40c1-4ade-b5b5-34890b40ea29 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:45.699511 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 194/390] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:11:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:45.700279 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a410201-40c1-4ade-b5b5-34890b40ea29 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:45.733990 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:45.772238 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94886724-771f-4d85-9d44-9158ea7da3ce None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:45.784444 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:45.802210 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a410201-40c1-4ade-b5b5-34890b40ea29 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:45.809610 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e920870d-ecae-44eb-b86f-faf1964bba6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:45.810246 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e920870d-ecae-44eb-b86f-faf1964bba6b None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:45.810524 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e920870d-ecae-44eb-b86f-faf1964bba6b None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:45.810794 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e920870d-ecae-44eb-b86f-faf1964bba6b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:45.821562 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e920870d-ecae-44eb-b86f-faf1964bba6b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:45.822996 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e920870d-ecae-44eb-b86f-faf1964bba6b None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:45.836705 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d11465e0-2545-4edb-b28b-f3bb3f8d7d0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:45.836953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-e920870d-ecae-44eb-b86f-faf1964bba6b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:45.836953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:45.836953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:45.836953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:45.836953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:45.836953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:45.836953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:45.836953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:45.836953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:45.836953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:45.836953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:45.836953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:45.836953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:11:45.836953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:11:45.836953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:11:45.836953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:11:45.838091 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:45.838091 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:45.838091 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:45.838091 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:45.838091 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:45.838091 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:45.838091 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:45.838091 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:45.838091 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:45.838091 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:45.838091 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:45.838091 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:45.838091 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:45.838091 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:45.838091 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:45.838091 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:45.838091 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:11:45.838928 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:11:45.838928 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:45.838928 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:45.839086 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11465e0-2545-4edb-b28b-f3bb3f8d7d0b None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:45.839086 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11465e0-2545-4edb-b28b-f3bb3f8d7d0b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:45.839086 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11465e0-2545-4edb-b28b-f3bb3f8d7d0b None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:45.839276 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 197/391] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:11:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:45.839342 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d11465e0-2545-4edb-b28b-f3bb3f8d7d0b None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:45.839342 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d11465e0-2545-4edb-b28b-f3bb3f8d7d0b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:45.840341 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e920870d-ecae-44eb-b86f-faf1964bba6b None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:45.840459 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d11465e0-2545-4edb-b28b-f3bb3f8d7d0b None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:45.858348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 195/392] 10.222.0.42 () {56 vars in 1137 bytes} [Thu Oct 27 01:11:45 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:45.858348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d11465e0-2545-4edb-b28b-f3bb3f8d7d0b None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:45.939301 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e920870d-ecae-44eb-b86f-faf1964bba6b None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:45.953107 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:45.957673 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d11465e0-2545-4edb-b28b-f3bb3f8d7d0b None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:45.975718 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af32b24-3fb0-45b0-828a-faba881c0810 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:45.976330 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af32b24-3fb0-45b0-828a-faba881c0810 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:45.976602 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af32b24-3fb0-45b0-828a-faba881c0810 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:45.976869 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af32b24-3fb0-45b0-828a-faba881c0810 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:45.984096 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af32b24-3fb0-45b0-828a-faba881c0810 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:45.985492 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af32b24-3fb0-45b0-828a-faba881c0810 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:45.991742 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f458672-f968-4c3b-b4dd-61f985fe5179 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:45.992364 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f458672-f968-4c3b-b4dd-61f985fe5179 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:45.992642 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f458672-f968-4c3b-b4dd-61f985fe5179 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:45.992914 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f458672-f968-4c3b-b4dd-61f985fe5179 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:45.996363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-5af32b24-3fb0-45b0-828a-faba881c0810 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 27 01:11:45.996363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:45.996363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:45.996363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:45.996363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:45.996363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:45.996363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:45.996363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:45.996363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:45.996363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:45.996363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:45.996363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:45.996363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 27 01:11:45.996363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 27 01:11:45.996363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 27 01:11:45.996363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 27 01:11:45.997411 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:45.997411 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:45.997411 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 27 01:11:45.997411 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:11:45.997411 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 27 01:11:45.997411 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:11:45.997411 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:45.997411 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:45.997411 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:45.997411 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:45.997411 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:45.997411 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:45.997411 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:45.997411 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:45.997411 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:45.997411 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:45.997411 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:45.997411 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:45.998392 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:45.998392 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:45.998392 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 27 01:11:45.998392 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 27 01:11:45.998392 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 27 01:11:45.998392 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 27 01:11:45.998392 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 27 01:11:45.998392 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 27 01:11:45.998392 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 27 01:11:45.998392 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:46.002011 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 198/393] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:11:45 2022] GET /identity/v3/users/cinder => generated 84 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:46.002817 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af32b24-3fb0-45b0-828a-faba881c0810 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:46.002994 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f458672-f968-4c3b-b4dd-61f985fe5179 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:46.005103 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f458672-f968-4c3b-b4dd-61f985fe5179 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:46.008026 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:46.022382 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-6f458672-f968-4c3b-b4dd-61f985fe5179 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:46.022382 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:46.022382 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:46.022382 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:46.022382 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:46.022382 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:46.022382 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:46.022382 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:46.022382 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:46.022382 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:46.022382 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:46.022382 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:46.022382 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:11:46.022382 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:11:46.022382 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:11:46.022382 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:11:46.023312 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:46.023312 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:46.023312 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:46.023312 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:46.023312 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:46.023312 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:46.023312 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:46.023312 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:46.023312 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:46.023312 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:46.023312 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:46.023312 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:46.023312 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:46.023312 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:46.023312 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:46.023312 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:46.023312 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:11:46.024269 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:11:46.024269 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:46.024269 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:46.024269 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 196/394] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:11:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:46.024269 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f458672-f968-4c3b-b4dd-61f985fe5179 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:46.104149 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af32b24-3fb0-45b0-828a-faba881c0810 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:46.123034 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f458672-f968-4c3b-b4dd-61f985fe5179 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:46.137794 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32acd04d-e9fe-4fd7-a767-048c5f390907 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:46.138408 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32acd04d-e9fe-4fd7-a767-048c5f390907 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:46.138679 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32acd04d-e9fe-4fd7-a767-048c5f390907 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:46.138944 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32acd04d-e9fe-4fd7-a767-048c5f390907 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:46.145509 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32acd04d-e9fe-4fd7-a767-048c5f390907 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:46.146932 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32acd04d-e9fe-4fd7-a767-048c5f390907 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:46.155991 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b58ef4-0dd7-4b08-a987-6212c369a5a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:46.156582 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b58ef4-0dd7-4b08-a987-6212c369a5a3 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:46.156857 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b58ef4-0dd7-4b08-a987-6212c369a5a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:46.157150 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b58ef4-0dd7-4b08-a987-6212c369a5a3 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:46.157780 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b58ef4-0dd7-4b08-a987-6212c369a5a3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:46.158079 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b58ef4-0dd7-4b08-a987-6212c369a5a3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:46.158177 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-32acd04d-e9fe-4fd7-a767-048c5f390907 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:11:46.158177 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:46.158177 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:46.158177 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:46.158177 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:46.158177 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:46.158177 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:46.158177 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:46.158177 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:46.158177 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:46.158177 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:46.158177 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:46.158177 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 27 01:11:46.158177 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 27 01:11:46.158177 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 27 01:11:46.159098 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 27 01:11:46.159098 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:46.159098 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:46.159098 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:46.159098 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:46.159098 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:46.159098 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:46.159098 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:46.159098 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:46.159098 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:46.159098 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:46.159098 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:46.159098 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:46.159098 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:46.159098 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:46.159098 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:46.159098 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:46.159956 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 27 01:11:46.159956 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 27 01:11:46.159956 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 27 01:11:46.159956 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 27 01:11:46.159956 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 27 01:11:46.159956 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 27 01:11:46.159956 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:11:46.159956 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:46.159956 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 199/395] 10.222.0.42 () {56 vars in 1111 bytes} [Thu Oct 27 01:11:46 2022] GET /identity/v3/projects/service => generated 88 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:46.160481 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b58ef4-0dd7-4b08-a987-6212c369a5a3 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:46.160555 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32acd04d-e9fe-4fd7-a767-048c5f390907 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:46.175284 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 197/396] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:11:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:46.176588 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b58ef4-0dd7-4b08-a987-6212c369a5a3 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:46.196465 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:46.231097 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:46.261414 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32acd04d-e9fe-4fd7-a767-048c5f390907 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:46.275829 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b58ef4-0dd7-4b08-a987-6212c369a5a3 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:46.294591 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a6dc0d-b2c4-481c-bb75-4d03c9f11e4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:46.295211 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a6dc0d-b2c4-481c-bb75-4d03c9f11e4e None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:46.295827 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a6dc0d-b2c4-481c-bb75-4d03c9f11e4e None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:46.295827 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a6dc0d-b2c4-481c-bb75-4d03c9f11e4e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:46.306799 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a6dc0d-b2c4-481c-bb75-4d03c9f11e4e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:46.308238 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a6dc0d-b2c4-481c-bb75-4d03c9f11e4e None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:46.309504 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3890fd4-c8fd-4728-8a43-d8687f7c9101 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:46.310100 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3890fd4-c8fd-4728-8a43-d8687f7c9101 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:46.310377 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3890fd4-c8fd-4728-8a43-d8687f7c9101 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:46.310646 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3890fd4-c8fd-4728-8a43-d8687f7c9101 None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:46.311361 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3890fd4-c8fd-4728-8a43-d8687f7c9101 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:46.311664 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3890fd4-c8fd-4728-8a43-d8687f7c9101 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:46.313065 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3890fd4-c8fd-4728-8a43-d8687f7c9101 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:46.322166 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-49a6dc0d-b2c4-481c-bb75-4d03c9f11e4e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:46.322166 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:46.322166 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:46.322166 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:46.322166 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:46.322166 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:46.322166 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:46.322166 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:46.322166 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:46.322166 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:46.322166 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:46.322166 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:46.322166 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:11:46.322166 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:11:46.322166 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:11:46.322166 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:11:46.323054 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:46.323054 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:46.323054 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:46.323054 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:46.323054 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:46.323054 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:46.323054 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:46.323054 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:46.323054 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:46.323054 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:46.323054 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:46.323054 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:46.323054 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:46.323054 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:46.323054 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:46.323054 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:46.323054 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:11:46.323891 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:11:46.323891 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:46.323891 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:46.323891 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 200/397] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:11:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:46.324569 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a6dc0d-b2c4-481c-bb75-4d03c9f11e4e None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:46.331100 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 198/398] 10.222.0.42 () {56 vars in 1137 bytes} [Thu Oct 27 01:11:46 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:46.331905 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3890fd4-c8fd-4728-8a43-d8687f7c9101 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:46.411540 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:46.427306 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a6dc0d-b2c4-481c-bb75-4d03c9f11e4e None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:46.429760 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3890fd4-c8fd-4728-8a43-d8687f7c9101 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:46.443201 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:46.461317 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ab7dd4-9c8a-462b-a623-10d6e6a112d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:46.461924 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ab7dd4-9c8a-462b-a623-10d6e6a112d9 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:46.462202 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ab7dd4-9c8a-462b-a623-10d6e6a112d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:46.462473 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ab7dd4-9c8a-462b-a623-10d6e6a112d9 None admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:46.463106 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2ab7dd4-9c8a-462b-a623-10d6e6a112d9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:46.463405 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2ab7dd4-9c8a-462b-a623-10d6e6a112d9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:46.464339 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a476ba-86fb-49a1-b1bd-fc5bd4fff84a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:46.464769 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2ab7dd4-9c8a-462b-a623-10d6e6a112d9 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:46.464933 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a476ba-86fb-49a1-b1bd-fc5bd4fff84a None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:46.465239 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a476ba-86fb-49a1-b1bd-fc5bd4fff84a None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:46.465509 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a476ba-86fb-49a1-b1bd-fc5bd4fff84a None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:46.466347 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a476ba-86fb-49a1-b1bd-fc5bd4fff84a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:46.466647 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a476ba-86fb-49a1-b1bd-fc5bd4fff84a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:46.468478 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a476ba-86fb-49a1-b1bd-fc5bd4fff84a None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:46.476826 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 201/399] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:11:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:46.477624 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ab7dd4-9c8a-462b-a623-10d6e6a112d9 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:46.487482 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 199/400] 10.222.0.42 () {56 vars in 1148 bytes} [Thu Oct 27 01:11:46 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:46.488154 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a476ba-86fb-49a1-b1bd-fc5bd4fff84a None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:46.574827 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ab7dd4-9c8a-462b-a623-10d6e6a112d9 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:46.590205 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a476ba-86fb-49a1-b1bd-fc5bd4fff84a None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:46.608555 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a613452-5168-44ad-831a-f4f8257ba7b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:46.609173 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a613452-5168-44ad-831a-f4f8257ba7b6 None admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:46.609446 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a613452-5168-44ad-831a-f4f8257ba7b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:46.609719 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a613452-5168-44ad-831a-f4f8257ba7b6 None admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:46.610350 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a613452-5168-44ad-831a-f4f8257ba7b6 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:46.611752 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a613452-5168-44ad-831a-f4f8257ba7b6 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:46.624643 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c30bb0-b2fe-4a29-9f6d-e9fdeff2460d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:46.625261 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c30bb0-b2fe-4a29-9f6d-e9fdeff2460d None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:46.625558 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c30bb0-b2fe-4a29-9f6d-e9fdeff2460d None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:46.625814 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c30bb0-b2fe-4a29-9f6d-e9fdeff2460d None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:46.626452 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c30bb0-b2fe-4a29-9f6d-e9fdeff2460d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:46.626752 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c30bb0-b2fe-4a29-9f6d-e9fdeff2460d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:46.628120 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c30bb0-b2fe-4a29-9f6d-e9fdeff2460d None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:46.634292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 202/401] 10.222.0.42 () {60 vars in 1132 bytes} [Thu Oct 27 01:11:46 2022] POST /identity/v3/users => generated 247 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:11:46.634292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a613452-5168-44ad-831a-f4f8257ba7b6 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:46.637822 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:46.640139 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 200/402] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:11:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:46.645963 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c30bb0-b2fe-4a29-9f6d-e9fdeff2460d None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:46.680422 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:46.762427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c30bb0-b2fe-4a29-9f6d-e9fdeff2460d None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:46.762858 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a613452-5168-44ad-831a-f4f8257ba7b6 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:46.806259 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb385ebe-c06f-4957-b068-73b523bbf5e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:46.808961 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed17a503-b89c-4069-81dd-9282369e776c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:46.809552 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb385ebe-c06f-4957-b068-73b523bbf5e1 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:46.809552 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb385ebe-c06f-4957-b068-73b523bbf5e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:46.809685 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed17a503-b89c-4069-81dd-9282369e776c None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:46.809770 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb385ebe-c06f-4957-b068-73b523bbf5e1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:46.810100 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed17a503-b89c-4069-81dd-9282369e776c None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:46.810100 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed17a503-b89c-4069-81dd-9282369e776c None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:46.819076 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed17a503-b89c-4069-81dd-9282369e776c None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:46.821343 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed17a503-b89c-4069-81dd-9282369e776c None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:46.823459 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb385ebe-c06f-4957-b068-73b523bbf5e1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:46.824976 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb385ebe-c06f-4957-b068-73b523bbf5e1 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:46.836526 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-ed17a503-b89c-4069-81dd-9282369e776c None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 27 01:11:46.836526 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:46.836526 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:46.836526 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:46.836526 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:46.836526 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:46.836526 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:46.836526 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:46.836526 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:46.836526 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:46.836526 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:46.836526 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:46.836526 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 27 01:11:46.836526 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 27 01:11:46.836526 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 27 01:11:46.836526 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 27 01:11:46.837472 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:46.837472 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:46.837472 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 27 01:11:46.837472 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:11:46.837472 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 27 01:11:46.837472 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:11:46.837472 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:46.837472 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:46.837472 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:46.837472 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:46.837472 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:46.837472 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:46.837472 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:46.837472 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:46.837472 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:46.837472 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:46.837472 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:46.837472 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:46.839192 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:46.839192 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:46.839192 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 27 01:11:46.839192 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 27 01:11:46.839192 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 27 01:11:46.839192 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 27 01:11:46.839192 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 27 01:11:46.839192 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 27 01:11:46.839192 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 27 01:11:46.839192 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:46.839192 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 201/403] 10.222.0.42 () {56 vars in 1102 bytes} [Thu Oct 27 01:11:46 2022] GET /identity/v3/users/neutron => generated 85 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:46.839192 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed17a503-b89c-4069-81dd-9282369e776c None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:46.839842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-fb385ebe-c06f-4957-b068-73b523bbf5e1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:46.839842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:46.839842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:46.839842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:46.839842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:46.839842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:46.839842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:46.839842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:46.839842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:46.839842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:46.839842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:46.839842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:46.839842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:11:46.839842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:11:46.839842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:11:46.839842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:11:46.840646 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:46.840646 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:46.840646 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:46.840646 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:46.840646 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:46.840646 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:46.840646 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:46.840646 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:46.840646 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:46.840646 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:46.840646 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:46.840646 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:46.840646 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:46.840646 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:46.840646 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:46.840646 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:46.840646 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:11:46.843928 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:11:46.843928 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:46.843928 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:46.843928 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 203/404] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:11:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:46.843928 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb385ebe-c06f-4957-b068-73b523bbf5e1 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:46.901028 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:46.940070 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:46.949647 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed17a503-b89c-4069-81dd-9282369e776c None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:46.951707 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb385ebe-c06f-4957-b068-73b523bbf5e1 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:46.985472 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e709c2b9-f8f0-4927-a832-0f87c60ed831 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:46.985672 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf6be63c-d039-4812-aa5a-284ee1bcce5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:46.986096 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e709c2b9-f8f0-4927-a832-0f87c60ed831 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:46.986299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6be63c-d039-4812-aa5a-284ee1bcce5e None admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:46.986402 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e709c2b9-f8f0-4927-a832-0f87c60ed831 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:46.986567 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6be63c-d039-4812-aa5a-284ee1bcce5e None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:46.986657 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e709c2b9-f8f0-4927-a832-0f87c60ed831 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:46.986833 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6be63c-d039-4812-aa5a-284ee1bcce5e None admin] PATH_INFO: `/v3/projects/23aad848d47c4d27ad4c138ff869e40f/users/51033d56f85f4282a49ded6338440677/roles/1b8f8dd9e8584053b048032db63254a4` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:46.993712 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e709c2b9-f8f0-4927-a832-0f87c60ed831 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:46.995148 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e709c2b9-f8f0-4927-a832-0f87c60ed831 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:47.002441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf6be63c-d039-4812-aa5a-284ee1bcce5e None admin] RBAC: Authorizing `identity:create_grant(project_id=23aad848d47c4d27ad4c138ff869e40f, user_id=51033d56f85f4282a49ded6338440677, role_id=1b8f8dd9e8584053b048032db63254a4)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:47.004107 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf6be63c-d039-4812-aa5a-284ee1bcce5e None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:47.006477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-e709c2b9-f8f0-4927-a832-0f87c60ed831 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:11:47.006477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:47.006477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:47.006477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:47.006477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:47.006477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:47.006477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:47.006477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:47.006477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:47.006477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:47.006477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:47.006477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:47.006477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 27 01:11:47.006477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 27 01:11:47.006477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 27 01:11:47.007292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 27 01:11:47.007292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:47.007292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:47.007292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:47.007292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:47.007292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:47.007292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:47.007292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:47.007292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:47.007292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:47.007292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:47.007292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:47.007292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:47.007292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:47.007292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:47.007292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:47.007292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:47.008132 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 27 01:11:47.008132 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 27 01:11:47.008132 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 27 01:11:47.008132 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 27 01:11:47.008132 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 27 01:11:47.008132 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 27 01:11:47.008132 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:11:47.008132 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:47.008132 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 204/405] 10.222.0.42 () {56 vars in 1111 bytes} [Thu Oct 27 01:11:46 2022] GET /identity/v3/projects/service => generated 88 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:47.008594 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e709c2b9-f8f0-4927-a832-0f87c60ed831 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:47.022724 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 202/406] 10.222.0.42 () {58 vars in 1439 bytes} [Thu Oct 27 01:11:46 2022] PUT /identity/v3/projects/23aad848d47c4d27ad4c138ff869e40f/users/51033d56f85f4282a49ded6338440677/roles/1b8f8dd9e8584053b048032db63254a4 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:11:47.023608 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf6be63c-d039-4812-aa5a-284ee1bcce5e None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:47.152598 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf6be63c-d039-4812-aa5a-284ee1bcce5e None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:47.163458 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e709c2b9-f8f0-4927-a832-0f87c60ed831 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:47.179572 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:47.199457 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3252d138-0a9d-4571-95eb-89183f116efd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:47.200451 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3252d138-0a9d-4571-95eb-89183f116efd None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:47.200875 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3252d138-0a9d-4571-95eb-89183f116efd None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:47.201348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3252d138-0a9d-4571-95eb-89183f116efd None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:47.202144 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3252d138-0a9d-4571-95eb-89183f116efd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:47.202612 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3252d138-0a9d-4571-95eb-89183f116efd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:47.204249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3252d138-0a9d-4571-95eb-89183f116efd None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:47.205738 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa0297b-7da3-4ab6-a8c0-e2bd3378575e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:47.206621 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa0297b-7da3-4ab6-a8c0-e2bd3378575e None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:47.207535 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa0297b-7da3-4ab6-a8c0-e2bd3378575e None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:47.207896 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa0297b-7da3-4ab6-a8c0-e2bd3378575e None admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:47.208970 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa0297b-7da3-4ab6-a8c0-e2bd3378575e None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:47.209392 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa0297b-7da3-4ab6-a8c0-e2bd3378575e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:47.210933 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa0297b-7da3-4ab6-a8c0-e2bd3378575e None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:47.219062 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 203/407] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:11:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:47.220120 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3252d138-0a9d-4571-95eb-89183f116efd None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:47.223219 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:47.259965 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 205/408] 10.222.0.42 () {56 vars in 1139 bytes} [Thu Oct 27 01:11:47 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:47.260634 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa0297b-7da3-4ab6-a8c0-e2bd3378575e None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:47.329773 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3252d138-0a9d-4571-95eb-89183f116efd None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:47.372317 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4699b9-d3ac-40bc-843f-8763fa3474d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:47.372925 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4699b9-d3ac-40bc-843f-8763fa3474d1 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:47.373276 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4699b9-d3ac-40bc-843f-8763fa3474d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:47.373555 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4699b9-d3ac-40bc-843f-8763fa3474d1 None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:47.374246 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4699b9-d3ac-40bc-843f-8763fa3474d1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:47.374550 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4699b9-d3ac-40bc-843f-8763fa3474d1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:47.375977 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4699b9-d3ac-40bc-843f-8763fa3474d1 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:47.390932 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa0297b-7da3-4ab6-a8c0-e2bd3378575e None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:47.393182 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 204/409] 10.222.0.42 () {56 vars in 1148 bytes} [Thu Oct 27 01:11:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:47.394101 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4699b9-d3ac-40bc-843f-8763fa3474d1 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:47.444074 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370f4465-3fae-440d-9168-d6af1b674dcc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:47.444996 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370f4465-3fae-440d-9168-d6af1b674dcc None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:47.445479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370f4465-3fae-440d-9168-d6af1b674dcc None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:47.445897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370f4465-3fae-440d-9168-d6af1b674dcc None admin] PATH_INFO: `/v3/projects/service` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:47.453651 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370f4465-3fae-440d-9168-d6af1b674dcc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:47.455546 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370f4465-3fae-440d-9168-d6af1b674dcc None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:47.468229 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-370f4465-3fae-440d-9168-d6af1b674dcc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:11:47.468229 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:47.468229 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:47.468229 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:47.468229 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:47.468229 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:47.468229 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:47.468229 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:47.468229 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:47.468229 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:47.468229 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:47.468229 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:47.468229 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 27 01:11:47.468229 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 27 01:11:47.468229 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 27 01:11:47.469303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 27 01:11:47.469303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:47.469303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:47.469303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:47.469303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:47.469303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:47.469303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:47.469303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:47.469303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:47.469303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:47.469303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:47.469303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:47.469303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:47.469303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:47.469303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:47.469303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:47.469303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:47.470309 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 27 01:11:47.470309 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 27 01:11:47.470309 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 27 01:11:47.470309 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 27 01:11:47.470309 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 27 01:11:47.470309 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 27 01:11:47.470309 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:11:47.470309 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:47.471045 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:47.473969 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 206/410] 10.222.0.42 () {56 vars in 1111 bytes} [Thu Oct 27 01:11:47 2022] GET /identity/v3/projects/service => generated 88 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:47.475132 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370f4465-3fae-440d-9168-d6af1b674dcc None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:47.510896 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4699b9-d3ac-40bc-843f-8763fa3474d1 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:47.512102 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:47.546540 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f6f6fa-2415-413f-a3bf-6b65783be45f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:47.547171 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f6f6fa-2415-413f-a3bf-6b65783be45f None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:47.547448 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f6f6fa-2415-413f-a3bf-6b65783be45f None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:47.547715 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f6f6fa-2415-413f-a3bf-6b65783be45f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:47.559027 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f6f6fa-2415-413f-a3bf-6b65783be45f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:47.559915 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f6f6fa-2415-413f-a3bf-6b65783be45f None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:47.575592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-54f6f6fa-2415-413f-a3bf-6b65783be45f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:47.575592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:47.575592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:47.575592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:47.575592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:47.575592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:47.575592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:47.575592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:47.575592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:47.575592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:47.575592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:47.575592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:47.575592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:11:47.575592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:11:47.575592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:11:47.575592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:11:47.576571 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:47.576571 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:47.576571 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:47.576571 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:47.576571 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:47.576571 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:47.576571 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:47.576571 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:47.576571 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:47.576571 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:47.576571 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:47.576571 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:47.576571 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:47.576571 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:47.576571 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:47.576571 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:47.576571 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:11:47.578080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:11:47.578080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:47.578080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:47.578080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 205/411] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:11:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:47.578080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f6f6fa-2415-413f-a3bf-6b65783be45f None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:47.596325 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370f4465-3fae-440d-9168-d6af1b674dcc None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:47.639865 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3ca808-20fb-475f-b5f6-fd8437a835a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:47.641017 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3ca808-20fb-475f-b5f6-fd8437a835a1 None admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:47.641529 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3ca808-20fb-475f-b5f6-fd8437a835a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:47.642020 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3ca808-20fb-475f-b5f6-fd8437a835a1 None admin] PATH_INFO: `/v3/projects/23aad848d47c4d27ad4c138ff869e40f/users/2880dbb00eec471285d242e8183dd8d0/roles/1b8f8dd9e8584053b048032db63254a4` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:47.659432 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac3ca808-20fb-475f-b5f6-fd8437a835a1 None admin] RBAC: Authorizing `identity:create_grant(project_id=23aad848d47c4d27ad4c138ff869e40f, user_id=2880dbb00eec471285d242e8183dd8d0, role_id=1b8f8dd9e8584053b048032db63254a4)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:47.661663 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac3ca808-20fb-475f-b5f6-fd8437a835a1 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:47.678366 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 207/412] 10.222.0.42 () {58 vars in 1439 bytes} [Thu Oct 27 01:11:47 2022] PUT /identity/v3/projects/23aad848d47c4d27ad4c138ff869e40f/users/2880dbb00eec471285d242e8183dd8d0/roles/1b8f8dd9e8584053b048032db63254a4 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:11:47.679651 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3ca808-20fb-475f-b5f6-fd8437a835a1 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:47.690911 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f6f6fa-2415-413f-a3bf-6b65783be45f None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:47.718657 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:47.759221 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67898829-b161-4a0e-92e9-ce224ce66841 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:47.759865 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67898829-b161-4a0e-92e9-ce224ce66841 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:47.760112 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67898829-b161-4a0e-92e9-ce224ce66841 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:47.760384 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67898829-b161-4a0e-92e9-ce224ce66841 None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:47.761266 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67898829-b161-4a0e-92e9-ce224ce66841 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:47.761441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67898829-b161-4a0e-92e9-ce224ce66841 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:47.762859 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67898829-b161-4a0e-92e9-ce224ce66841 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:47.785860 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:47.793238 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 206/413] 10.222.0.42 () {56 vars in 1148 bytes} [Thu Oct 27 01:11:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:47.794258 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67898829-b161-4a0e-92e9-ce224ce66841 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:47.811081 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3ca808-20fb-475f-b5f6-fd8437a835a1 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:47.854629 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb99d0c-964a-48b3-a131-df84c86191d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:47.855758 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb99d0c-964a-48b3-a131-df84c86191d8 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:47.856305 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb99d0c-964a-48b3-a131-df84c86191d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:47.856808 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb99d0c-964a-48b3-a131-df84c86191d8 None admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:47.857709 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb99d0c-964a-48b3-a131-df84c86191d8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:47.858257 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb99d0c-964a-48b3-a131-df84c86191d8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:47.860091 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb99d0c-964a-48b3-a131-df84c86191d8 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:47.874201 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 208/414] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:11:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:47.881118 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb99d0c-964a-48b3-a131-df84c86191d8 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:47.940318 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67898829-b161-4a0e-92e9-ce224ce66841 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:48.001990 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83cefc4-853b-467d-b943-ad6908c398c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:48.002839 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83cefc4-853b-467d-b943-ad6908c398c4 None admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:48.004805 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83cefc4-853b-467d-b943-ad6908c398c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:48.005469 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83cefc4-853b-467d-b943-ad6908c398c4 None admin] PATH_INFO: `/v3/projects/23aad848d47c4d27ad4c138ff869e40f/users/d35a3d40a1924064859562a72bab0f6d/roles/1b8f8dd9e8584053b048032db63254a4` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:48.027505 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c83cefc4-853b-467d-b943-ad6908c398c4 None admin] RBAC: Authorizing `identity:create_grant(project_id=23aad848d47c4d27ad4c138ff869e40f, user_id=d35a3d40a1924064859562a72bab0f6d, role_id=1b8f8dd9e8584053b048032db63254a4)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:48.029765 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c83cefc4-853b-467d-b943-ad6908c398c4 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:48.054934 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 207/415] 10.222.0.42 () {58 vars in 1439 bytes} [Thu Oct 27 01:11:47 2022] PUT /identity/v3/projects/23aad848d47c4d27ad4c138ff869e40f/users/d35a3d40a1924064859562a72bab0f6d/roles/1b8f8dd9e8584053b048032db63254a4 => generated 0 bytes in 261 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:11:48.076080 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb99d0c-964a-48b3-a131-df84c86191d8 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:48.135968 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba202d66-473c-4ba4-8081-56cc0b345154 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:48.136604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba202d66-473c-4ba4-8081-56cc0b345154 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:48.136875 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba202d66-473c-4ba4-8081-56cc0b345154 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:48.137164 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba202d66-473c-4ba4-8081-56cc0b345154 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:48.147815 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba202d66-473c-4ba4-8081-56cc0b345154 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:48.149295 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba202d66-473c-4ba4-8081-56cc0b345154 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:48.167907 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-ba202d66-473c-4ba4-8081-56cc0b345154 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:11:48.167907 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:48.167907 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:48.167907 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:48.167907 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:48.167907 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:48.167907 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:48.167907 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:48.167907 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:48.167907 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:48.167907 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:48.167907 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:48.167907 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 27 01:11:48.167907 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 27 01:11:48.167907 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 27 01:11:48.168993 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 27 01:11:48.168993 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:48.168993 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:48.168993 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:48.168993 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:48.168993 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:48.168993 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:48.168993 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:48.168993 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:48.168993 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:48.168993 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:48.168993 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:48.168993 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:48.168993 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:48.168993 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:48.168993 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:48.168993 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:48.170053 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 27 01:11:48.170053 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 27 01:11:48.170053 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 27 01:11:48.170053 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 27 01:11:48.170053 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 27 01:11:48.170053 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 27 01:11:48.170053 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:11:48.170053 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:48.170053 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 209/416] 10.222.0.42 () {56 vars in 1111 bytes} [Thu Oct 27 01:11:47 2022] GET /identity/v3/projects/service => generated 88 bytes in 290 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:48.176631 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83cefc4-853b-467d-b943-ad6908c398c4 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:48.222358 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:48.328594 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83cefc4-853b-467d-b943-ad6908c398c4 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:48.371699 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e1340c-122a-48e8-a89a-e79ff62f0b16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:48.372316 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e1340c-122a-48e8-a89a-e79ff62f0b16 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:48.372594 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e1340c-122a-48e8-a89a-e79ff62f0b16 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:48.372867 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e1340c-122a-48e8-a89a-e79ff62f0b16 None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:48.373598 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e1340c-122a-48e8-a89a-e79ff62f0b16 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:48.373901 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e1340c-122a-48e8-a89a-e79ff62f0b16 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:48.375387 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e1340c-122a-48e8-a89a-e79ff62f0b16 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:48.387766 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 208/417] 10.222.0.42 () {56 vars in 1148 bytes} [Thu Oct 27 01:11:48 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:48.393868 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba202d66-473c-4ba4-8081-56cc0b345154 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:48.403669 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:48.543320 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba202d66-473c-4ba4-8081-56cc0b345154 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:48.587926 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c075c153-7ae9-411e-ade3-418984a4e2ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:48.588502 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c075c153-7ae9-411e-ade3-418984a4e2ab None admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:48.588772 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c075c153-7ae9-411e-ade3-418984a4e2ab None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:48.589038 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c075c153-7ae9-411e-ade3-418984a4e2ab None admin] PATH_INFO: `/v3/projects/23aad848d47c4d27ad4c138ff869e40f/users/4072417990c94b0488871332d28224ba/roles/1b8f8dd9e8584053b048032db63254a4` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:48.611877 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c075c153-7ae9-411e-ade3-418984a4e2ab None admin] RBAC: Authorizing `identity:create_grant(project_id=23aad848d47c4d27ad4c138ff869e40f, user_id=4072417990c94b0488871332d28224ba, role_id=1b8f8dd9e8584053b048032db63254a4)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:48.613876 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c075c153-7ae9-411e-ade3-418984a4e2ab None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:48.641548 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 210/418] 10.222.0.42 () {58 vars in 1439 bytes} [Thu Oct 27 01:11:48 2022] PUT /identity/v3/projects/23aad848d47c4d27ad4c138ff869e40f/users/4072417990c94b0488871332d28224ba/roles/1b8f8dd9e8584053b048032db63254a4 => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:11:49.351780 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311a5c01-a3d7-43dd-8165-e4f2a5124f6d None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:49.352337 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311a5c01-a3d7-43dd-8165-e4f2a5124f6d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:49.352420 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311a5c01-a3d7-43dd-8165-e4f2a5124f6d None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:49.354165 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 209/419] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:49.360885 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ac82e7-8066-41d1-af45-d7e2d20d1f3f None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:49.361435 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ac82e7-8066-41d1-af45-d7e2d20d1f3f None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:49.361435 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ac82e7-8066-41d1-af45-d7e2d20d1f3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:49.382000 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-37ac82e7-8066-41d1-af45-d7e2d20d1f3f None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:49.383210 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-37ac82e7-8066-41d1-af45-d7e2d20d1f3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:49.433344 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:49.464002 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 211/420] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:11:49.558633 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058e9f83-afbb-40f8-a5f9-d42cd3905f6b None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:49.558914 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058e9f83-afbb-40f8-a5f9-d42cd3905f6b None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:49.559188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058e9f83-afbb-40f8-a5f9-d42cd3905f6b None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:49.560772 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 210/421] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:49.566155 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdea0f13-74f6-4ec0-baf3-959794906764 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:49.567344 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdea0f13-74f6-4ec0-baf3-959794906764 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:49.567734 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdea0f13-74f6-4ec0-baf3-959794906764 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:49.591976 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-cdea0f13-74f6-4ec0-baf3-959794906764 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:49.593179 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-cdea0f13-74f6-4ec0-baf3-959794906764 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:49.684925 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 212/422] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:11:49.930096 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-058e9f83-afbb-40f8-a5f9-d42cd3905f6b None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:50.039100 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdea0f13-74f6-4ec0-baf3-959794906764 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:50.040835 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:50.067217 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-058e9f83-afbb-40f8-a5f9-d42cd3905f6b None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:50.119660 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cabdeea5-c765-4aed-acc0-4ad955806f36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:50.120920 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabdeea5-c765-4aed-acc0-4ad955806f36 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:50.121376 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabdeea5-c765-4aed-acc0-4ad955806f36 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:50.121790 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabdeea5-c765-4aed-acc0-4ad955806f36 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:50.148149 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cabdeea5-c765-4aed-acc0-4ad955806f36 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:50.154266 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-cabdeea5-c765-4aed-acc0-4ad955806f36 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 27 01:11:50.154266 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:50.154266 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:50.154266 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:50.154266 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:50.154266 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:50.154266 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:50.154266 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:50.154266 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:50.154266 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:50.154266 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:50.154266 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:50.154266 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:50.154266 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:50.154266 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:50.154266 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:50.155157 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:50.155157 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:50.155157 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:50.155157 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:50.155157 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:50.155157 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:50.155157 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:50.155157 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:50.155157 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:50.155157 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:50.155157 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:50.155157 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:50.155157 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:50.155157 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:50.155157 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:50.155157 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:50.155967 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:50.155967 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:50.155967 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:50.155967 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:50.155967 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:50.155967 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:50.155967 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:50.155967 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:50.155967 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:50.155967 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:50.155967 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 27 01:11:50.155967 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:50.157026 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 211/423] 10.222.0.42 () {56 vars in 1102 bytes} [Thu Oct 27 01:11:49 2022] GET /identity/v3/roles/service => generated 85 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:50.163377 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cabdeea5-c765-4aed-acc0-4ad955806f36 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:50.235783 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdea0f13-74f6-4ec0-baf3-959794906764 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:50.264078 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:50.281777 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cabdeea5-c765-4aed-acc0-4ad955806f36 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:50.321239 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c19733-23d3-4cd9-8210-14083c0c30ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:50.322235 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c19733-23d3-4cd9-8210-14083c0c30ed None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:50.322703 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c19733-23d3-4cd9-8210-14083c0c30ed None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:50.323167 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c19733-23d3-4cd9-8210-14083c0c30ed None admin] PATH_INFO: `/v3/roles/service` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:50.336196 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c19733-23d3-4cd9-8210-14083c0c30ed None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:50.341897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-60c19733-23d3-4cd9-8210-14083c0c30ed None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 27 01:11:50.341897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:50.341897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:50.341897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:50.341897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:50.341897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:50.341897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:50.341897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:50.341897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:50.341897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:50.341897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:50.341897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:50.341897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:50.341897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:50.341897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:50.341897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:50.342848 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:50.342848 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:50.342848 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:50.342848 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:50.342848 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:50.342848 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:50.342848 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:50.342848 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:50.342848 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:50.342848 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:50.342848 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:50.342848 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:50.342848 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:50.342848 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:50.342848 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:50.342848 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:50.343660 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:50.343660 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:50.343660 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:50.343660 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:50.343660 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:50.343660 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:50.343660 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:50.343660 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:50.343660 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:50.343660 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:50.343660 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 27 01:11:50.343660 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:50.346414 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 213/424] 10.222.0.42 () {56 vars in 1102 bytes} [Thu Oct 27 01:11:50 2022] GET /identity/v3/roles/service => generated 85 bytes in 308 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 27 01:11:50.352975 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c19733-23d3-4cd9-8210-14083c0c30ed None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:50.361230 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c47abb-e056-43af-b3d4-20e0fe580a00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:50.365363 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c47abb-e056-43af-b3d4-20e0fe580a00 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:50.366816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c47abb-e056-43af-b3d4-20e0fe580a00 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:50.367330 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c47abb-e056-43af-b3d4-20e0fe580a00 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:50.371034 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c47abb-e056-43af-b3d4-20e0fe580a00 None admin] RBAC: Adding query filter params (name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:50.371656 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c47abb-e056-43af-b3d4-20e0fe580a00 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:50.373339 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c47abb-e056-43af-b3d4-20e0fe580a00 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:50.391975 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 212/425] 10.222.0.42 () {56 vars in 1103 bytes} [Thu Oct 27 01:11:50 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:11:50.410748 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c47abb-e056-43af-b3d4-20e0fe580a00 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:50.442684 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:50.539626 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c19733-23d3-4cd9-8210-14083c0c30ed None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:50.577815 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c47abb-e056-43af-b3d4-20e0fe580a00 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:50.579006 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e118a180-a053-4014-b0c9-afc9aad436e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:50.580047 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e118a180-a053-4014-b0c9-afc9aad436e6 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:50.580520 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e118a180-a053-4014-b0c9-afc9aad436e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:50.581000 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e118a180-a053-4014-b0c9-afc9aad436e6 None admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:50.582460 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e118a180-a053-4014-b0c9-afc9aad436e6 None admin] RBAC: Adding query filter params (name=service) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:50.582978 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e118a180-a053-4014-b0c9-afc9aad436e6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:50.584591 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e118a180-a053-4014-b0c9-afc9aad436e6 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:50.593516 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 214/426] 10.222.0.42 () {56 vars in 1103 bytes} [Thu Oct 27 01:11:50 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:50.595114 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a21f7a5-913f-47da-b1fb-ef594676a02b None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:50.595611 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a21f7a5-913f-47da-b1fb-ef594676a02b None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:50.596076 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a21f7a5-913f-47da-b1fb-ef594676a02b None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:50.597750 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 215/427] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:50.599208 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2954220-ecb9-49b5-8b28-529b05c3e174 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:50.599684 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2954220-ecb9-49b5-8b28-529b05c3e174 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:50.600167 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2954220-ecb9-49b5-8b28-529b05c3e174 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:50.601861 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 216/428] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:50.602846 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2954220-ecb9-49b5-8b28-529b05c3e174 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:50.615823 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a47a98b-029e-4085-a946-2f3cdce30c7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:50.616452 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a47a98b-029e-4085-a946-2f3cdce30c7d None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:50.616727 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a47a98b-029e-4085-a946-2f3cdce30c7d None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:50.617039 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a47a98b-029e-4085-a946-2f3cdce30c7d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:50.629053 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a47a98b-029e-4085-a946-2f3cdce30c7d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:50.630507 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a47a98b-029e-4085-a946-2f3cdce30c7d None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:50.641626 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:50.645682 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-6a47a98b-029e-4085-a946-2f3cdce30c7d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:50.645682 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:50.645682 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:50.645682 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:50.645682 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:50.645682 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:50.645682 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:50.645682 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:50.645682 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:50.645682 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:50.645682 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:50.645682 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:50.645682 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:11:50.645682 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:11:50.645682 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:11:50.645682 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:11:50.646552 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:50.646552 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:50.646552 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:50.646552 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:50.646552 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:50.646552 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:50.646552 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:50.646552 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:50.646552 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:50.646552 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:50.646552 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:50.646552 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:50.646552 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:50.646552 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:50.646552 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:50.646552 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:50.646552 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:11:50.647395 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:11:50.647395 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:50.647395 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:50.647395 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 213/429] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:11:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 27 01:11:50.648010 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a72ec8-dc7c-4d9e-a444-5dc76804ead3 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:50.648261 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a72ec8-dc7c-4d9e-a444-5dc76804ead3 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:50.648545 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a72ec8-dc7c-4d9e-a444-5dc76804ead3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:50.667026 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-09a72ec8-dc7c-4d9e-a444-5dc76804ead3 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:50.668386 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-09a72ec8-dc7c-4d9e-a444-5dc76804ead3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:50.718974 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2954220-ecb9-49b5-8b28-529b05c3e174 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:50.730635 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:50.757089 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 214/430] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:11:50.757089 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870adae2-8293-4f21-99eb-c3c989f02ade None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:50.757089 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870adae2-8293-4f21-99eb-c3c989f02ade None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:50.757089 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870adae2-8293-4f21-99eb-c3c989f02ade None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:50.766983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a4669c-d7ef-43af-8bfe-f96c456f0fae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:50.767619 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a4669c-d7ef-43af-8bfe-f96c456f0fae None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:50.767900 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a4669c-d7ef-43af-8bfe-f96c456f0fae None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:50.768167 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a4669c-d7ef-43af-8bfe-f96c456f0fae None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:50.774927 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-870adae2-8293-4f21-99eb-c3c989f02ade None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:50.776511 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-870adae2-8293-4f21-99eb-c3c989f02ade None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:50.779861 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0a4669c-d7ef-43af-8bfe-f96c456f0fae None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:50.781346 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0a4669c-d7ef-43af-8bfe-f96c456f0fae None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:50.796291 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-f0a4669c-d7ef-43af-8bfe-f96c456f0fae None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:50.796291 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:50.796291 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:50.796291 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:50.796291 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:50.796291 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:50.796291 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:50.796291 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:50.796291 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:50.796291 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:50.796291 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:50.796291 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:50.796291 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:11:50.796291 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:11:50.796291 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:11:50.796291 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:11:50.797267 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:50.797267 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:50.797267 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:50.797267 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:50.797267 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:50.797267 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:50.797267 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:50.797267 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:50.797267 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:50.797267 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:50.797267 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:50.797267 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:50.797267 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:50.797267 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:50.797267 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:50.797267 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:50.797267 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:11:50.798487 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:11:50.798487 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:50.798487 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:50.798487 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 217/431] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:11:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:50.803717 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a4669c-d7ef-43af-8bfe-f96c456f0fae None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:50.860587 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 215/432] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:11:50.866836 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870adae2-8293-4f21-99eb-c3c989f02ade None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:50.945423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:50.968157 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a4669c-d7ef-43af-8bfe-f96c456f0fae None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:51.034617 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef031c1-c21f-4609-97a3-7252251fce9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:51.035745 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef031c1-c21f-4609-97a3-7252251fce9b None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:51.036023 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef031c1-c21f-4609-97a3-7252251fce9b None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:51.036291 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef031c1-c21f-4609-97a3-7252251fce9b None admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:51.036929 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cef031c1-c21f-4609-97a3-7252251fce9b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:51.037248 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cef031c1-c21f-4609-97a3-7252251fce9b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:51.038663 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cef031c1-c21f-4609-97a3-7252251fce9b None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:51.056080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870adae2-8293-4f21-99eb-c3c989f02ade None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:51.057549 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 218/433] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:11:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:51.058458 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef031c1-c21f-4609-97a3-7252251fce9b None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:51.076427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:51.095705 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b5be78-28e4-4148-826f-99dd30e311cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:51.096568 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b5be78-28e4-4148-826f-99dd30e311cf None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:51.097088 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b5be78-28e4-4148-826f-99dd30e311cf None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:51.097969 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b5be78-28e4-4148-826f-99dd30e311cf None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:51.098713 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b5be78-28e4-4148-826f-99dd30e311cf None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:51.099122 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b5be78-28e4-4148-826f-99dd30e311cf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:51.100571 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b5be78-28e4-4148-826f-99dd30e311cf None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:51.113285 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 216/434] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:11:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:51.114205 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b5be78-28e4-4148-826f-99dd30e311cf None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:51.185562 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef031c1-c21f-4609-97a3-7252251fce9b None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:51.221524 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3811f68b-4925-44fc-be05-c36945825586 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:51.222174 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3811f68b-4925-44fc-be05-c36945825586 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:51.223046 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3811f68b-4925-44fc-be05-c36945825586 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:51.223317 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3811f68b-4925-44fc-be05-c36945825586 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:51.235763 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3811f68b-4925-44fc-be05-c36945825586 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:51.240905 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b5be78-28e4-4148-826f-99dd30e311cf None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:51.241139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-3811f68b-4925-44fc-be05-c36945825586 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 27 01:11:51.241139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:51.241139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:51.241139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:51.241139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:51.241139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:51.241139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:51.241139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:51.241139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:51.241139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:51.241139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:51.241139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:51.241139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:51.241139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:51.241139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:51.241139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:51.242203 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:51.242203 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:51.242203 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:51.242203 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:51.242203 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:51.242203 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:51.242203 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:51.242203 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:51.242203 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:51.242203 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:51.242203 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:51.242203 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:51.242203 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:51.242203 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:51.242203 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:51.242203 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:51.243079 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:51.243079 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:51.243079 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:51.243079 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:51.243079 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:51.243079 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:51.243079 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:51.243079 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:51.243079 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:51.243079 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:51.243079 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 27 01:11:51.243079 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:51.243079 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 219/435] 10.222.0.42 () {56 vars in 1102 bytes} [Thu Oct 27 01:11:51 2022] GET /identity/v3/roles/service => generated 85 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:51.243810 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3811f68b-4925-44fc-be05-c36945825586 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:51.254394 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:51.285810 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a8918a-8885-44da-b458-d4c4b1dc3c3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:51.286438 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a8918a-8885-44da-b458-d4c4b1dc3c3d None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:51.286707 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a8918a-8885-44da-b458-d4c4b1dc3c3d None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:51.286970 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a8918a-8885-44da-b458-d4c4b1dc3c3d None admin] PATH_INFO: `/v3/users/swift` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:51.296111 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a8918a-8885-44da-b458-d4c4b1dc3c3d None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:51.297715 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a8918a-8885-44da-b458-d4c4b1dc3c3d None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:51.310464 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-f7a8918a-8885-44da-b458-d4c4b1dc3c3d None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 27 01:11:51.310464 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:51.310464 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:51.310464 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:51.310464 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:51.310464 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:51.310464 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:51.310464 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:51.310464 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:51.310464 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:51.310464 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:51.310464 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:51.310464 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 27 01:11:51.310464 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 27 01:11:51.310464 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 27 01:11:51.310464 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 27 01:11:51.311593 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:51.311593 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:51.311593 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 27 01:11:51.311593 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:11:51.311593 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 27 01:11:51.311593 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:11:51.311593 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:51.311593 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:51.311593 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:51.311593 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:51.311593 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:51.311593 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:51.311593 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:51.311593 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:51.311593 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:51.311593 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:51.311593 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:51.311593 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:51.312739 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:51.312739 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:51.312739 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 27 01:11:51.312739 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 27 01:11:51.312739 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 27 01:11:51.312739 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 27 01:11:51.312739 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 27 01:11:51.312739 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 27 01:11:51.312739 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 27 01:11:51.312739 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:51.320311 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 217/436] 10.222.0.42 () {56 vars in 1096 bytes} [Thu Oct 27 01:11:51 2022] GET /identity/v3/users/swift => generated 83 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:51.320311 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a8918a-8885-44da-b458-d4c4b1dc3c3d None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:51.355584 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3811f68b-4925-44fc-be05-c36945825586 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:51.359543 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:51.389221 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac5a6eb4-90c5-4877-93a8-4489d4306011 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:51.389770 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5a6eb4-90c5-4877-93a8-4489d4306011 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:51.390061 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5a6eb4-90c5-4877-93a8-4489d4306011 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:51.390321 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5a6eb4-90c5-4877-93a8-4489d4306011 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:51.397746 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac5a6eb4-90c5-4877-93a8-4489d4306011 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:51.399198 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac5a6eb4-90c5-4877-93a8-4489d4306011 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:51.410200 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-ac5a6eb4-90c5-4877-93a8-4489d4306011 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 27 01:11:51.410200 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:51.410200 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:51.410200 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:51.410200 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:51.410200 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:51.410200 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:51.410200 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:51.410200 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:51.410200 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:51.410200 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:51.410200 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:51.410200 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 27 01:11:51.410200 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 27 01:11:51.410200 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 27 01:11:51.410200 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 27 01:11:51.411115 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:51.411115 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:51.411115 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 27 01:11:51.411115 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:11:51.411115 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 27 01:11:51.411115 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:11:51.411115 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:51.411115 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:51.411115 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:51.411115 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:51.411115 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:51.411115 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:51.411115 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:51.411115 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:51.411115 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:51.411115 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:51.411115 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:51.411115 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:51.412009 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:51.412009 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:51.412009 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 27 01:11:51.412009 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 27 01:11:51.412009 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 27 01:11:51.412009 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 27 01:11:51.412009 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 27 01:11:51.412009 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 27 01:11:51.412009 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 27 01:11:51.412009 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:51.412849 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 220/437] 10.222.0.42 () {56 vars in 1093 bytes} [Thu Oct 27 01:11:51 2022] GET /identity/v3/users/nova => generated 82 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:51.413687 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac5a6eb4-90c5-4877-93a8-4489d4306011 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:51.421034 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a8918a-8885-44da-b458-d4c4b1dc3c3d None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:51.455715 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97e979d-4b02-4fd9-8e37-e790f65474bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:51.456350 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97e979d-4b02-4fd9-8e37-e790f65474bd None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:51.456627 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97e979d-4b02-4fd9-8e37-e790f65474bd None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:51.457480 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97e979d-4b02-4fd9-8e37-e790f65474bd None admin] PATH_INFO: `/v3/roles/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:51.467361 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c97e979d-4b02-4fd9-8e37-e790f65474bd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:51.472793 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-c97e979d-4b02-4fd9-8e37-e790f65474bd None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 27 01:11:51.472793 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:51.472793 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:51.472793 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:51.472793 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:51.472793 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:51.472793 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:51.472793 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:51.472793 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:51.472793 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:51.472793 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:51.472793 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:51.472793 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:51.472793 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:51.472793 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:51.472793 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:51.473730 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:51.473730 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:51.473730 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:51.473730 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:51.473730 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:51.473730 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:51.473730 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:51.473730 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:51.473730 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:51.473730 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:51.473730 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:51.473730 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:51.473730 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:51.473730 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:51.473730 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:51.473730 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:51.474512 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:51.474512 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:51.474512 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:51.474512 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:51.474512 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:51.474512 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:51.474512 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:51.474512 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:51.474512 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:51.474512 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:51.474512 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 27 01:11:51.474512 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:51.474512 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 218/438] 10.222.0.42 () {56 vars in 1102 bytes} [Thu Oct 27 01:11:51 2022] GET /identity/v3/roles/service => generated 85 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:51.475167 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97e979d-4b02-4fd9-8e37-e790f65474bd None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:51.479899 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:51.515208 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac5a6eb4-90c5-4877-93a8-4489d4306011 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:51.547671 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec88fec6-f120-47e2-902c-7358609ad991 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:51.548285 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec88fec6-f120-47e2-902c-7358609ad991 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:51.548523 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec88fec6-f120-47e2-902c-7358609ad991 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:51.548788 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec88fec6-f120-47e2-902c-7358609ad991 None admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:51.549472 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec88fec6-f120-47e2-902c-7358609ad991 None admin] RBAC: Adding query filter params (name=service) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:51.549765 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec88fec6-f120-47e2-902c-7358609ad991 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:51.551149 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec88fec6-f120-47e2-902c-7358609ad991 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:51.559282 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 221/439] 10.222.0.42 () {56 vars in 1103 bytes} [Thu Oct 27 01:11:51 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:51.560246 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d8ddfc-b5e6-48a1-9f61-4e80e4df3d19 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:51.560523 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d8ddfc-b5e6-48a1-9f61-4e80e4df3d19 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:51.560787 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d8ddfc-b5e6-48a1-9f61-4e80e4df3d19 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:51.562015 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 222/440] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:51 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:51.562692 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d8ddfc-b5e6-48a1-9f61-4e80e4df3d19 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:51.572824 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:51.576865 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97e979d-4b02-4fd9-8e37-e790f65474bd None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:51.612454 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d4ad7f-6b92-4c4d-81d7-68ed43e64a63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:51.613339 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d4ad7f-6b92-4c4d-81d7-68ed43e64a63 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:51.613339 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d4ad7f-6b92-4c4d-81d7-68ed43e64a63 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:51.613668 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d4ad7f-6b92-4c4d-81d7-68ed43e64a63 None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:51.614370 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06d4ad7f-6b92-4c4d-81d7-68ed43e64a63 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:51.614671 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06d4ad7f-6b92-4c4d-81d7-68ed43e64a63 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:51.616092 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06d4ad7f-6b92-4c4d-81d7-68ed43e64a63 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:51.632699 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 219/441] 10.222.0.42 () {56 vars in 1135 bytes} [Thu Oct 27 01:11:51 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:51.633611 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d4ad7f-6b92-4c4d-81d7-68ed43e64a63 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:51.662710 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d8ddfc-b5e6-48a1-9f61-4e80e4df3d19 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:51.688270 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:51.695968 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c832714-f501-45bd-b143-07e7d0b343b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:51.696566 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c832714-f501-45bd-b143-07e7d0b343b6 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:51.696867 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c832714-f501-45bd-b143-07e7d0b343b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:51.697166 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c832714-f501-45bd-b143-07e7d0b343b6 None admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:51.697881 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c832714-f501-45bd-b143-07e7d0b343b6 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:51.698185 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c832714-f501-45bd-b143-07e7d0b343b6 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:51.699608 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c832714-f501-45bd-b143-07e7d0b343b6 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:51.717098 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 223/442] 10.222.0.42 () {56 vars in 1133 bytes} [Thu Oct 27 01:11:51 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:51.717961 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c832714-f501-45bd-b143-07e7d0b343b6 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:51.731022 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d4ad7f-6b92-4c4d-81d7-68ed43e64a63 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:51.764213 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4521376-bc08-466d-ba12-71c1d45e3391 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:51.764810 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4521376-bc08-466d-ba12-71c1d45e3391 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:51.765105 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4521376-bc08-466d-ba12-71c1d45e3391 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:51.765380 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4521376-bc08-466d-ba12-71c1d45e3391 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:51.766038 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4521376-bc08-466d-ba12-71c1d45e3391 None admin] RBAC: Adding query filter params (name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:51.766336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4521376-bc08-466d-ba12-71c1d45e3391 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:51.767696 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4521376-bc08-466d-ba12-71c1d45e3391 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:51.775601 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 220/443] 10.222.0.42 () {56 vars in 1103 bytes} [Thu Oct 27 01:11:51 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:51.776712 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3700e7d-4b9e-40d0-8a72-a69a82c65c1c None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:51.777049 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3700e7d-4b9e-40d0-8a72-a69a82c65c1c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:51.777355 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3700e7d-4b9e-40d0-8a72-a69a82c65c1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:51.796132 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-b3700e7d-4b9e-40d0-8a72-a69a82c65c1c None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:51.797375 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-b3700e7d-4b9e-40d0-8a72-a69a82c65c1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:51.812754 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:51.822917 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c832714-f501-45bd-b143-07e7d0b343b6 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:51.856027 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519a0331-e285-477c-8750-23bf12d496d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:51.856642 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519a0331-e285-477c-8750-23bf12d496d7 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:51.856915 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519a0331-e285-477c-8750-23bf12d496d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:51.857213 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519a0331-e285-477c-8750-23bf12d496d7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:51.866516 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 221/444] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:11:51.867340 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3700e7d-4b9e-40d0-8a72-a69a82c65c1c None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:51.867514 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519a0331-e285-477c-8750-23bf12d496d7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:51.869020 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519a0331-e285-477c-8750-23bf12d496d7 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:51.884188 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-519a0331-e285-477c-8750-23bf12d496d7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:51.884188 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:51.884188 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:51.884188 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:51.884188 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:51.884188 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:51.884188 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:51.884188 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:51.884188 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:51.884188 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:51.884188 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:51.884188 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:51.884188 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:11:51.884188 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:11:51.884188 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:11:51.884188 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:11:51.885316 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:51.885316 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:51.885316 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:51.885316 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:51.885316 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:51.885316 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:51.885316 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:51.885316 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:51.885316 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:51.885316 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:51.885316 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:51.885316 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:51.885316 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:51.885316 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:51.885316 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:51.885316 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:51.885316 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:11:51.886352 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:11:51.886352 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:51.886352 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:51.886352 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 224/445] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:11:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:51.890697 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519a0331-e285-477c-8750-23bf12d496d7 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:51.931889 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:51.975934 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3700e7d-4b9e-40d0-8a72-a69a82c65c1c None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:51.993691 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519a0331-e285-477c-8750-23bf12d496d7 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:52.012707 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53ee9cd-50ab-486c-a20e-22e56ffe79e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:52.013350 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53ee9cd-50ab-486c-a20e-22e56ffe79e2 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:52.013624 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53ee9cd-50ab-486c-a20e-22e56ffe79e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:52.013891 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53ee9cd-50ab-486c-a20e-22e56ffe79e2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:52.025541 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f53ee9cd-50ab-486c-a20e-22e56ffe79e2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:52.026959 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f53ee9cd-50ab-486c-a20e-22e56ffe79e2 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:52.027888 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3b8b60-0e43-4191-97aa-319870e93de2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:52.028722 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3b8b60-0e43-4191-97aa-319870e93de2 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:52.029077 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3b8b60-0e43-4191-97aa-319870e93de2 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:52.029721 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3b8b60-0e43-4191-97aa-319870e93de2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:52.035778 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:52.042553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-f53ee9cd-50ab-486c-a20e-22e56ffe79e2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:52.042553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:52.042553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:52.042553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:52.042553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:52.042553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:52.042553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:52.042553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:52.042553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:52.042553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:52.042553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:52.042553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:52.042553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:11:52.042553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:11:52.042553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:11:52.042553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:11:52.043664 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:52.043664 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:52.043664 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:52.043664 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:52.043664 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:52.043664 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:52.043664 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:52.043664 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:52.043664 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:52.043664 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:52.043664 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:52.043664 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:52.043664 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:52.043664 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:52.043664 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:52.043664 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:52.043664 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:11:52.044754 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:11:52.044754 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:52.044754 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:52.046543 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 222/446] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:11:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:52.052712 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53ee9cd-50ab-486c-a20e-22e56ffe79e2 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:52.106188 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f3b8b60-0e43-4191-97aa-319870e93de2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:52.107757 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f3b8b60-0e43-4191-97aa-319870e93de2 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:52.148025 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:52.164003 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53ee9cd-50ab-486c-a20e-22e56ffe79e2 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:52.197887 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6221d1fd-eea0-4c8e-8aa2-417ead692c25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:52.198431 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6221d1fd-eea0-4c8e-8aa2-417ead692c25 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:52.198705 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6221d1fd-eea0-4c8e-8aa2-417ead692c25 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:52.198973 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6221d1fd-eea0-4c8e-8aa2-417ead692c25 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:52.209369 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6221d1fd-eea0-4c8e-8aa2-417ead692c25 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:52.209984 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 225/447] 10.222.0.42 () {58 vars in 1282 bytes} [Thu Oct 27 01:11:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 27 01:11:52.210887 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6221d1fd-eea0-4c8e-8aa2-417ead692c25 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:52.211281 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3b8b60-0e43-4191-97aa-319870e93de2 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:52.227165 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-6221d1fd-eea0-4c8e-8aa2-417ead692c25 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:52.227165 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:52.227165 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:52.227165 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:52.227165 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:52.227165 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:52.227165 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:52.227165 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:52.227165 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:52.227165 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:52.227165 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:52.227165 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:52.227165 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:11:52.227165 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:11:52.227165 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:11:52.227165 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:11:52.228250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:52.228250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:52.228250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:52.228250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:52.228250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:52.228250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:52.228250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:52.228250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:52.228250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:52.228250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:52.228250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:52.228250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:52.228250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:52.228250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:52.228250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:52.228250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:52.228250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:11:52.229120 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:11:52.229120 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:52.229120 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:52.229120 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 223/448] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:11:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:52.229120 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6221d1fd-eea0-4c8e-8aa2-417ead692c25 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:52.280269 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:52.312342 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3b8b60-0e43-4191-97aa-319870e93de2 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:52.327715 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6221d1fd-eea0-4c8e-8aa2-417ead692c25 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:52.345853 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8267bd7-411f-464b-ac9e-18bae394538f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:52.346449 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8267bd7-411f-464b-ac9e-18bae394538f None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:52.346728 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8267bd7-411f-464b-ac9e-18bae394538f None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:52.346999 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8267bd7-411f-464b-ac9e-18bae394538f None admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:52.347890 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8267bd7-411f-464b-ac9e-18bae394538f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:52.348200 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8267bd7-411f-464b-ac9e-18bae394538f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:52.349611 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8267bd7-411f-464b-ac9e-18bae394538f None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:52.356024 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:52.361295 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 226/449] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:11:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:52.362138 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8267bd7-411f-464b-ac9e-18bae394538f None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:52.362239 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c8dc4f-ea4f-438d-8908-89b268817a62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:52.362755 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c8dc4f-ea4f-438d-8908-89b268817a62 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:52.363537 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c8dc4f-ea4f-438d-8908-89b268817a62 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:52.364332 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c8dc4f-ea4f-438d-8908-89b268817a62 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:52.364975 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c8dc4f-ea4f-438d-8908-89b268817a62 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:52.365312 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c8dc4f-ea4f-438d-8908-89b268817a62 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:52.366662 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c8dc4f-ea4f-438d-8908-89b268817a62 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:52.378086 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 224/450] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:11:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:52.380555 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c8dc4f-ea4f-438d-8908-89b268817a62 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:52.461502 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8267bd7-411f-464b-ac9e-18bae394538f None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:52.478748 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c8dc4f-ea4f-438d-8908-89b268817a62 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:52.492341 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:52.494458 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072adc53-8e5c-4d1d-92c1-09899555e204 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:52.495105 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072adc53-8e5c-4d1d-92c1-09899555e204 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:52.495385 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072adc53-8e5c-4d1d-92c1-09899555e204 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:52.495655 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072adc53-8e5c-4d1d-92c1-09899555e204 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:52.506288 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-072adc53-8e5c-4d1d-92c1-09899555e204 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:52.511472 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-072adc53-8e5c-4d1d-92c1-09899555e204 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 27 01:11:52.511472 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:52.511472 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:52.511472 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:52.511472 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:52.511472 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:52.511472 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:52.511472 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:52.511472 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:52.511472 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:52.511472 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:52.511472 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:52.511472 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:52.511472 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:52.511472 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:52.511472 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:52.512412 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:52.512412 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:52.512412 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:52.512412 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:52.512412 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:52.512412 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:52.512412 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:52.512412 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:52.512412 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:52.512412 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:52.512412 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:52.512412 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:52.512412 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:52.512412 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:52.512412 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:52.512412 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:52.513336 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:52.513336 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:52.513336 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:52.513336 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:52.513336 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:52.513336 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:52.513336 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:52.513336 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:52.513336 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:52.513336 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:52.513336 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 27 01:11:52.513336 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:52.513336 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 227/451] 10.222.0.42 () {56 vars in 1102 bytes} [Thu Oct 27 01:11:52 2022] GET /identity/v3/roles/service => generated 85 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:52.513980 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072adc53-8e5c-4d1d-92c1-09899555e204 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:52.514050 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29be9c2c-9e2c-489b-8dfa-107a34769a5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:52.514599 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29be9c2c-9e2c-489b-8dfa-107a34769a5b None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:52.514933 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29be9c2c-9e2c-489b-8dfa-107a34769a5b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:52.515233 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29be9c2c-9e2c-489b-8dfa-107a34769a5b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:52.525799 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29be9c2c-9e2c-489b-8dfa-107a34769a5b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:52.527239 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29be9c2c-9e2c-489b-8dfa-107a34769a5b None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:52.541476 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-29be9c2c-9e2c-489b-8dfa-107a34769a5b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:52.541476 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:52.541476 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:52.541476 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:52.541476 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:52.541476 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:52.541476 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:52.541476 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:52.541476 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:52.541476 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:52.541476 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:52.541476 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:52.541476 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:11:52.541476 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:11:52.541476 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:11:52.541476 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:11:52.542383 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:52.542383 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:52.542383 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:52.542383 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:52.542383 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:52.542383 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:52.542383 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:52.542383 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:52.542383 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:52.542383 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:52.542383 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:52.542383 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:52.542383 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:52.542383 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:52.542383 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:52.542383 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:52.542383 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:11:52.543222 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:11:52.543222 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:52.543222 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:52.543222 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 225/452] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:11:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:52.543431 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29be9c2c-9e2c-489b-8dfa-107a34769a5b None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:52.577527 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:52.618871 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072adc53-8e5c-4d1d-92c1-09899555e204 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:52.640732 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29be9c2c-9e2c-489b-8dfa-107a34769a5b None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:52.652813 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519fe040-0374-4ba9-82d0-dcffeb72f710 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:52.653518 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519fe040-0374-4ba9-82d0-dcffeb72f710 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:52.653791 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519fe040-0374-4ba9-82d0-dcffeb72f710 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:52.654065 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519fe040-0374-4ba9-82d0-dcffeb72f710 None admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:52.654687 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519fe040-0374-4ba9-82d0-dcffeb72f710 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:52.654983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519fe040-0374-4ba9-82d0-dcffeb72f710 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:52.656379 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519fe040-0374-4ba9-82d0-dcffeb72f710 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:52.669317 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 228/453] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:11:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:52.670101 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519fe040-0374-4ba9-82d0-dcffeb72f710 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:52.674409 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b68f5a3-4aa8-4317-8b35-9a4f3333541e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:52.675022 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b68f5a3-4aa8-4317-8b35-9a4f3333541e None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:52.675294 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b68f5a3-4aa8-4317-8b35-9a4f3333541e None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:52.675628 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b68f5a3-4aa8-4317-8b35-9a4f3333541e None admin] PATH_INFO: `/v3/users/glance` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:52.684292 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b68f5a3-4aa8-4317-8b35-9a4f3333541e None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:52.685750 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b68f5a3-4aa8-4317-8b35-9a4f3333541e None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:52.696928 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-0b68f5a3-4aa8-4317-8b35-9a4f3333541e None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 27 01:11:52.696928 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:52.696928 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:52.696928 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:52.696928 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:52.696928 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:52.696928 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:52.696928 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:52.696928 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:52.696928 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:52.696928 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:52.696928 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:52.696928 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 27 01:11:52.696928 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 27 01:11:52.696928 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 27 01:11:52.696928 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 27 01:11:52.698091 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:52.698091 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:52.698091 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 27 01:11:52.698091 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:11:52.698091 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 27 01:11:52.698091 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:11:52.698091 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:52.698091 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:52.698091 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:52.698091 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:52.698091 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:52.698091 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:52.698091 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:52.698091 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:52.698091 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:52.698091 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:52.698091 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:52.698091 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:52.698972 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:52.698972 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:52.698972 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 27 01:11:52.698972 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 27 01:11:52.698972 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 27 01:11:52.698972 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 27 01:11:52.698972 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 27 01:11:52.698972 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 27 01:11:52.698972 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 27 01:11:52.698972 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:52.698972 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 226/454] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:11:52 2022] GET /identity/v3/users/glance => generated 84 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:52.698972 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b68f5a3-4aa8-4317-8b35-9a4f3333541e None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:52.725199 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:52.770161 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519fe040-0374-4ba9-82d0-dcffeb72f710 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:52.787765 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:52.798032 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b68f5a3-4aa8-4317-8b35-9a4f3333541e None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:52.808285 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a822007f-d95e-4cd2-b457-0c315df5ba25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:52.808913 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a822007f-d95e-4cd2-b457-0c315df5ba25 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:52.809243 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a822007f-d95e-4cd2-b457-0c315df5ba25 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:52.809510 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a822007f-d95e-4cd2-b457-0c315df5ba25 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:52.819267 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a822007f-d95e-4cd2-b457-0c315df5ba25 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:52.820700 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a822007f-d95e-4cd2-b457-0c315df5ba25 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:52.831727 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-a822007f-d95e-4cd2-b457-0c315df5ba25 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:11:52.831727 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:52.831727 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:52.831727 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:52.831727 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:52.831727 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:52.831727 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:52.831727 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:52.831727 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:52.831727 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:52.831727 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:52.831727 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:52.831727 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 27 01:11:52.831727 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 27 01:11:52.831727 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 27 01:11:52.833363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 27 01:11:52.833363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:52.833363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:52.833363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:52.833363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:52.833363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:52.833363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:52.833363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:52.833363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:52.833363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:52.833363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:52.833363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:52.833363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:52.833363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:52.833363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:52.833363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:52.833363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:52.834702 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 27 01:11:52.834702 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 27 01:11:52.834702 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 27 01:11:52.834702 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 27 01:11:52.834702 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 27 01:11:52.834702 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 27 01:11:52.834702 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:11:52.834702 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:52.834702 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 229/455] 10.222.0.42 () {56 vars in 1111 bytes} [Thu Oct 27 01:11:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:52.834702 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a822007f-d95e-4cd2-b457-0c315df5ba25 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:52.839521 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdbcc959-7266-4bbb-b698-df9e7e79c21a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:52.840130 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbcc959-7266-4bbb-b698-df9e7e79c21a None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:52.840407 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbcc959-7266-4bbb-b698-df9e7e79c21a None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:52.840671 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbcc959-7266-4bbb-b698-df9e7e79c21a None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:52.841363 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdbcc959-7266-4bbb-b698-df9e7e79c21a None admin] RBAC: Adding query filter params (name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:52.841660 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdbcc959-7266-4bbb-b698-df9e7e79c21a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:52.843436 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdbcc959-7266-4bbb-b698-df9e7e79c21a None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:52.857382 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 227/456] 10.222.0.42 () {56 vars in 1103 bytes} [Thu Oct 27 01:11:52 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:52.857382 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdbcc959-7266-4bbb-b698-df9e7e79c21a None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:52.934494 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a822007f-d95e-4cd2-b457-0c315df5ba25 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:52.937980 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:52.953715 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdbcc959-7266-4bbb-b698-df9e7e79c21a None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:52.967334 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7061a8-e15d-4bfa-bc8d-74b984134805 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:52.967930 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7061a8-e15d-4bfa-bc8d-74b984134805 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:52.968204 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7061a8-e15d-4bfa-bc8d-74b984134805 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:52.968493 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7061a8-e15d-4bfa-bc8d-74b984134805 None admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:52.969126 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be7061a8-e15d-4bfa-bc8d-74b984134805 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:52.969426 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be7061a8-e15d-4bfa-bc8d-74b984134805 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:52.970781 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be7061a8-e15d-4bfa-bc8d-74b984134805 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:52.981956 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 230/457] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:11:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:52.982664 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7061a8-e15d-4bfa-bc8d-74b984134805 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:52.990274 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c6ca5e-7223-4a66-a0a4-b236f7bf2731 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:52.990899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c6ca5e-7223-4a66-a0a4-b236f7bf2731 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:52.991176 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c6ca5e-7223-4a66-a0a4-b236f7bf2731 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:52.991455 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c6ca5e-7223-4a66-a0a4-b236f7bf2731 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:52.999158 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c6ca5e-7223-4a66-a0a4-b236f7bf2731 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:53.000619 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c6ca5e-7223-4a66-a0a4-b236f7bf2731 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:53.012616 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-58c6ca5e-7223-4a66-a0a4-b236f7bf2731 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 27 01:11:53.012616 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:53.012616 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:53.012616 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:53.012616 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:53.012616 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:53.012616 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:53.012616 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:53.012616 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:53.012616 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:53.012616 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:53.012616 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:53.012616 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 27 01:11:53.012616 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 27 01:11:53.012616 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 27 01:11:53.012616 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 27 01:11:53.013585 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:53.013585 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:53.013585 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 27 01:11:53.013585 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:11:53.013585 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 27 01:11:53.013585 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:11:53.013585 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:53.013585 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:53.013585 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:53.013585 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:53.013585 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:53.013585 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:53.013585 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:53.013585 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:53.013585 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:53.013585 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:53.013585 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:53.013585 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:53.014477 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:53.014477 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:53.014477 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 27 01:11:53.014477 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 27 01:11:53.014477 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 27 01:11:53.014477 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 27 01:11:53.014477 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 27 01:11:53.014477 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 27 01:11:53.014477 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 27 01:11:53.014477 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:53.018299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 228/458] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:11:52 2022] GET /identity/v3/users/cinder => generated 84 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:53.018998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c6ca5e-7223-4a66-a0a4-b236f7bf2731 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:53.021438 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:53.081212 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7061a8-e15d-4bfa-bc8d-74b984134805 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:53.114330 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d03d249-309b-43c5-8576-2c2ca8639619 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:53.114930 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d03d249-309b-43c5-8576-2c2ca8639619 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:53.115208 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d03d249-309b-43c5-8576-2c2ca8639619 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:53.115480 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d03d249-309b-43c5-8576-2c2ca8639619 None admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:53.116192 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d03d249-309b-43c5-8576-2c2ca8639619 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:53.116494 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d03d249-309b-43c5-8576-2c2ca8639619 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:53.117312 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c6ca5e-7223-4a66-a0a4-b236f7bf2731 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:53.117949 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d03d249-309b-43c5-8576-2c2ca8639619 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:53.135090 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 231/459] 10.222.0.42 () {56 vars in 1137 bytes} [Thu Oct 27 01:11:52 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:53.135880 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d03d249-309b-43c5-8576-2c2ca8639619 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:53.150919 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9438562-eca9-4fe8-828d-415db6cf08ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:53.151499 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9438562-eca9-4fe8-828d-415db6cf08ac None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:53.151775 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9438562-eca9-4fe8-828d-415db6cf08ac None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:53.152038 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9438562-eca9-4fe8-828d-415db6cf08ac None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:53.152717 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9438562-eca9-4fe8-828d-415db6cf08ac None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:53.153013 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9438562-eca9-4fe8-828d-415db6cf08ac None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:53.154448 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9438562-eca9-4fe8-828d-415db6cf08ac None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:53.166574 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 229/460] 10.222.0.42 () {56 vars in 1148 bytes} [Thu Oct 27 01:11:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:53.167357 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9438562-eca9-4fe8-828d-415db6cf08ac None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:53.177972 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:53.230869 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:53.236896 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d03d249-309b-43c5-8576-2c2ca8639619 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:53.267982 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9438562-eca9-4fe8-828d-415db6cf08ac None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:53.270604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d35087-c17c-4f8f-84ca-74f441816f09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:53.271259 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d35087-c17c-4f8f-84ca-74f441816f09 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:53.271535 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d35087-c17c-4f8f-84ca-74f441816f09 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:53.271805 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d35087-c17c-4f8f-84ca-74f441816f09 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:53.281437 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d35087-c17c-4f8f-84ca-74f441816f09 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:53.282877 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d35087-c17c-4f8f-84ca-74f441816f09 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:53.298579 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-28d35087-c17c-4f8f-84ca-74f441816f09 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:53.298579 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:53.298579 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:53.298579 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:53.298579 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:53.298579 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:53.298579 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:53.298579 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:53.298579 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:53.298579 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:53.298579 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:53.298579 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:53.298579 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:11:53.298579 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:11:53.298579 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:11:53.298579 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:11:53.299650 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:53.299650 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:53.299650 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:53.299650 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:53.299650 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:53.299650 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:53.299650 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:53.299650 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:53.299650 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:53.299650 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:53.299650 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:53.299650 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:53.299650 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:53.299650 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:53.299650 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:53.299650 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:53.299650 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:11:53.300782 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:11:53.300782 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:53.300782 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:53.300782 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 232/461] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:11:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:53.301094 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d35087-c17c-4f8f-84ca-74f441816f09 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:53.307056 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3feeb662-5483-4584-b39d-590028e831aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:53.307689 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3feeb662-5483-4584-b39d-590028e831aa None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:53.307966 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3feeb662-5483-4584-b39d-590028e831aa None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:53.308236 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3feeb662-5483-4584-b39d-590028e831aa None admin] PATH_INFO: `/v3/projects/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:53.314934 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3feeb662-5483-4584-b39d-590028e831aa None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:53.316921 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3feeb662-5483-4584-b39d-590028e831aa None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:53.328996 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-3feeb662-5483-4584-b39d-590028e831aa None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:11:53.328996 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:53.328996 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:53.328996 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:53.328996 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:53.328996 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:53.328996 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:53.328996 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:53.328996 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:53.328996 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:53.328996 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:53.328996 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:53.328996 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 27 01:11:53.328996 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 27 01:11:53.328996 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 27 01:11:53.330056 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 27 01:11:53.330056 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:53.330056 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:53.330056 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:53.330056 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:53.330056 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:53.330056 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:53.330056 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:53.330056 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:53.330056 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:53.330056 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:53.330056 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:53.330056 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:53.330056 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:53.330056 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:53.330056 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:53.330056 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:53.331120 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 27 01:11:53.331120 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 27 01:11:53.331120 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 27 01:11:53.331120 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 27 01:11:53.331120 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 27 01:11:53.331120 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 27 01:11:53.331120 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:11:53.331120 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:53.331120 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 230/462] 10.222.0.42 () {56 vars in 1111 bytes} [Thu Oct 27 01:11:53 2022] GET /identity/v3/projects/service => generated 88 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:53.331120 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3feeb662-5483-4584-b39d-590028e831aa None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:53.398716 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:53.405267 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d35087-c17c-4f8f-84ca-74f441816f09 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:53.432353 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3feeb662-5483-4584-b39d-590028e831aa None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:53.438055 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48da073a-7585-49b7-83cb-e5572c2b2ee0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:53.438649 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48da073a-7585-49b7-83cb-e5572c2b2ee0 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:53.438924 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48da073a-7585-49b7-83cb-e5572c2b2ee0 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:53.439193 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48da073a-7585-49b7-83cb-e5572c2b2ee0 None admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:53.439911 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48da073a-7585-49b7-83cb-e5572c2b2ee0 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:53.440209 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48da073a-7585-49b7-83cb-e5572c2b2ee0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:53.442249 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48da073a-7585-49b7-83cb-e5572c2b2ee0 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:53.459189 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 233/463] 10.222.0.42 () {56 vars in 1137 bytes} [Thu Oct 27 01:11:53 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:53.459961 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48da073a-7585-49b7-83cb-e5572c2b2ee0 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:53.466534 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11f301c3-f29f-4de1-8f57-1ff0487754b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:53.467331 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f301c3-f29f-4de1-8f57-1ff0487754b1 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:53.467602 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f301c3-f29f-4de1-8f57-1ff0487754b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:53.467868 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f301c3-f29f-4de1-8f57-1ff0487754b1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:53.473225 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:53.535478 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11f301c3-f29f-4de1-8f57-1ff0487754b1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:53.537484 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11f301c3-f29f-4de1-8f57-1ff0487754b1 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:53.560527 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48da073a-7585-49b7-83cb-e5572c2b2ee0 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:53.593973 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b11267-c23a-4c6b-883b-57d761d49c60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:53.594605 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b11267-c23a-4c6b-883b-57d761d49c60 None admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:53.594875 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b11267-c23a-4c6b-883b-57d761d49c60 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:53.595142 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b11267-c23a-4c6b-883b-57d761d49c60 None admin] PATH_INFO: `/v3/projects/23aad848d47c4d27ad4c138ff869e40f/users/77f2cf9e0c3541f9bccfb7e213d88609/roles/1b8f8dd9e8584053b048032db63254a4` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:53.599239 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:53.609336 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09b11267-c23a-4c6b-883b-57d761d49c60 None admin] RBAC: Authorizing `identity:create_grant(project_id=23aad848d47c4d27ad4c138ff869e40f, user_id=77f2cf9e0c3541f9bccfb7e213d88609, role_id=1b8f8dd9e8584053b048032db63254a4)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:53.610762 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09b11267-c23a-4c6b-883b-57d761d49c60 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:53.623360 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 234/464] 10.222.0.42 () {58 vars in 1439 bytes} [Thu Oct 27 01:11:53 2022] PUT /identity/v3/projects/23aad848d47c4d27ad4c138ff869e40f/users/77f2cf9e0c3541f9bccfb7e213d88609/roles/1b8f8dd9e8584053b048032db63254a4 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:11:53.624234 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b11267-c23a-4c6b-883b-57d761d49c60 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:53.640020 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 231/465] 10.222.0.42 () {58 vars in 1282 bytes} [Thu Oct 27 01:11:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 27 01:11:53.641235 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11f301c3-f29f-4de1-8f57-1ff0487754b1 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:53.701035 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:53.751767 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b11267-c23a-4c6b-883b-57d761d49c60 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:53.762718 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11f301c3-f29f-4de1-8f57-1ff0487754b1 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:53.790180 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe1cf57a-7604-42a1-80e8-b78ce218a48a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:53.790749 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1cf57a-7604-42a1-80e8-b78ce218a48a None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:53.794485 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1cf57a-7604-42a1-80e8-b78ce218a48a None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:53.794792 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1cf57a-7604-42a1-80e8-b78ce218a48a None admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:53.795924 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe1cf57a-7604-42a1-80e8-b78ce218a48a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:53.796306 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe1cf57a-7604-42a1-80e8-b78ce218a48a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:53.798188 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe1cf57a-7604-42a1-80e8-b78ce218a48a None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:53.806731 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a57f53d-9d1f-4e5d-935e-eb7b31170fbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:53.807344 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a57f53d-9d1f-4e5d-935e-eb7b31170fbd None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:53.807621 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a57f53d-9d1f-4e5d-935e-eb7b31170fbd None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:53.807890 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a57f53d-9d1f-4e5d-935e-eb7b31170fbd None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:53.808592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a57f53d-9d1f-4e5d-935e-eb7b31170fbd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:53.808898 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a57f53d-9d1f-4e5d-935e-eb7b31170fbd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:53.812000 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a57f53d-9d1f-4e5d-935e-eb7b31170fbd None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:53.813824 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 235/466] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:11:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:53.814680 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe1cf57a-7604-42a1-80e8-b78ce218a48a None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:53.826026 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 232/467] 10.222.0.42 () {56 vars in 1148 bytes} [Thu Oct 27 01:11:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:53.827148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a57f53d-9d1f-4e5d-935e-eb7b31170fbd None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:53.854667 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:53.926201 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe1cf57a-7604-42a1-80e8-b78ce218a48a None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:53.929881 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a57f53d-9d1f-4e5d-935e-eb7b31170fbd None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:53.943271 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:53.964868 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23edab40-18b8-4fb6-b21a-2d1e6d91ff8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:53.965534 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23edab40-18b8-4fb6-b21a-2d1e6d91ff8f None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:53.965817 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23edab40-18b8-4fb6-b21a-2d1e6d91ff8f None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:53.966102 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23edab40-18b8-4fb6-b21a-2d1e6d91ff8f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:53.967169 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b292f3-0701-454f-a9d7-19cd818b074f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:53.967792 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b292f3-0701-454f-a9d7-19cd818b074f None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:53.968065 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b292f3-0701-454f-a9d7-19cd818b074f None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:53.968332 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b292f3-0701-454f-a9d7-19cd818b074f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:53.979162 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b292f3-0701-454f-a9d7-19cd818b074f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:53.980598 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b292f3-0701-454f-a9d7-19cd818b074f None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:53.994670 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-77b292f3-0701-454f-a9d7-19cd818b074f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:53.994670 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:53.994670 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:53.994670 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:53.994670 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:53.994670 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:53.994670 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:53.994670 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:53.994670 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:53.994670 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:53.994670 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:53.994670 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:53.994670 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:11:53.994670 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:11:53.994670 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:11:53.994670 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:11:53.995589 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:53.995589 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:53.995589 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:53.995589 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:53.995589 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:53.995589 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:53.995589 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:53.995589 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:53.995589 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:53.995589 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:53.995589 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:53.995589 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:53.995589 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:53.995589 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:53.995589 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:53.995589 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:53.995589 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:11:53.996489 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:11:53.996489 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:53.996489 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:54.000327 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 233/468] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:11:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:54.000968 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b292f3-0701-454f-a9d7-19cd818b074f None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:54.048906 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23edab40-18b8-4fb6-b21a-2d1e6d91ff8f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:54.050307 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23edab40-18b8-4fb6-b21a-2d1e6d91ff8f None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:54.084849 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:54.110315 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b292f3-0701-454f-a9d7-19cd818b074f None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:54.145849 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e661a4f-eb88-4479-b72b-8619c304fd9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:54.146634 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e661a4f-eb88-4479-b72b-8619c304fd9a None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:54.147006 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e661a4f-eb88-4479-b72b-8619c304fd9a None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:54.147369 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e661a4f-eb88-4479-b72b-8619c304fd9a None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:54.155355 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e661a4f-eb88-4479-b72b-8619c304fd9a None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:54.156872 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e661a4f-eb88-4479-b72b-8619c304fd9a None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:54.163340 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 236/469] 10.222.0.42 () {58 vars in 1282 bytes} [Thu Oct 27 01:11:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 27 01:11:54.163340 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23edab40-18b8-4fb6-b21a-2d1e6d91ff8f None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:54.174249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-6e661a4f-eb88-4479-b72b-8619c304fd9a None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 27 01:11:54.174249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:54.174249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:54.174249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:54.174249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:54.174249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:54.174249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:54.174249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:54.174249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:54.174249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:54.174249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:54.174249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:54.174249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 27 01:11:54.174249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 27 01:11:54.174249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 27 01:11:54.174249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 27 01:11:54.175368 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:54.175368 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:54.175368 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 27 01:11:54.175368 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:11:54.175368 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 27 01:11:54.175368 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:11:54.175368 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:54.175368 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:54.175368 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:54.175368 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:54.175368 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:54.175368 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:54.175368 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:54.175368 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:54.175368 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:54.175368 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:54.175368 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:54.175368 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:54.176875 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:54.176875 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:54.176875 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 27 01:11:54.176875 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 27 01:11:54.176875 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 27 01:11:54.176875 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 27 01:11:54.176875 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 27 01:11:54.176875 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 27 01:11:54.176875 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 27 01:11:54.176875 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:54.176875 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 234/470] 10.222.0.42 () {56 vars in 1102 bytes} [Thu Oct 27 01:11:54 2022] GET /identity/v3/users/neutron => generated 85 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:54.176875 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e661a4f-eb88-4479-b72b-8619c304fd9a None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:54.184356 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:54.269821 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23edab40-18b8-4fb6-b21a-2d1e6d91ff8f None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:54.286765 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e661a4f-eb88-4479-b72b-8619c304fd9a None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:54.308789 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf65c487-9cb1-452a-8522-18403846324c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:54.309658 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf65c487-9cb1-452a-8522-18403846324c None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:54.309935 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf65c487-9cb1-452a-8522-18403846324c None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:54.310295 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf65c487-9cb1-452a-8522-18403846324c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:54.310905 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf65c487-9cb1-452a-8522-18403846324c None admin] RBAC: Adding query filter params (role.id=1b8f8dd9e8584053b048032db63254a4, scope.project.id=23aad848d47c4d27ad4c138ff869e40f, user.id=51033d56f85f4282a49ded6338440677) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:54.311206 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf65c487-9cb1-452a-8522-18403846324c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:54.312640 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf65c487-9cb1-452a-8522-18403846324c None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:54.324172 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d2d7af-d456-45a3-af02-2720516de200 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:54.324883 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d2d7af-d456-45a3-af02-2720516de200 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:54.325270 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d2d7af-d456-45a3-af02-2720516de200 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:54.325604 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d2d7af-d456-45a3-af02-2720516de200 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:54.326361 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d2d7af-d456-45a3-af02-2720516de200 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:54.326696 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d2d7af-d456-45a3-af02-2720516de200 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:54.328160 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d2d7af-d456-45a3-af02-2720516de200 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:54.329470 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 237/471] 10.222.0.42 () {56 vars in 1374 bytes} [Thu Oct 27 01:11:54 2022] GET /identity/v3/role_assignments?user.id=51033d56f85f4282a49ded6338440677&scope.project.id=23aad848d47c4d27ad4c138ff869e40f&role.id=1b8f8dd9e8584053b048032db63254a4 => generated 608 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:54.330366 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf65c487-9cb1-452a-8522-18403846324c None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:54.339833 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:54.351773 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 235/472] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:11:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:54.353401 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d2d7af-d456-45a3-af02-2720516de200 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:54.457222 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:54.481529 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf65c487-9cb1-452a-8522-18403846324c None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:54.483358 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d2d7af-d456-45a3-af02-2720516de200 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:54.526164 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d4bf61-1507-49f0-b673-4c0e42607de1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:54.526808 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d4bf61-1507-49f0-b673-4c0e42607de1 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:54.527089 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d4bf61-1507-49f0-b673-4c0e42607de1 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:54.527358 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d4bf61-1507-49f0-b673-4c0e42607de1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:54.534662 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e49870-b775-45a1-bd4f-624a249cb099 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:54.535565 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e49870-b775-45a1-bd4f-624a249cb099 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:54.535565 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e49870-b775-45a1-bd4f-624a249cb099 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:54.536568 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e49870-b775-45a1-bd4f-624a249cb099 None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:54.536834 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e49870-b775-45a1-bd4f-624a249cb099 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:54.537175 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e49870-b775-45a1-bd4f-624a249cb099 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:54.539451 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e49870-b775-45a1-bd4f-624a249cb099 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:54.542468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d4bf61-1507-49f0-b673-4c0e42607de1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:54.544052 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d4bf61-1507-49f0-b673-4c0e42607de1 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:54.559023 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 236/473] 10.222.0.42 () {56 vars in 1139 bytes} [Thu Oct 27 01:11:54 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:54.559795 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e49870-b775-45a1-bd4f-624a249cb099 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:54.561466 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-e1d4bf61-1507-49f0-b673-4c0e42607de1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:54.561466 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:54.561466 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:54.561466 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:54.561466 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:54.561466 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:54.561466 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:54.561466 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:54.561466 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:54.561466 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:54.561466 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:54.561466 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:54.561466 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:11:54.561466 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:11:54.561466 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:11:54.561466 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:11:54.562374 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:54.562374 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:54.562374 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:54.562374 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:54.562374 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:54.562374 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:54.562374 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:54.562374 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:54.562374 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:54.562374 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:54.562374 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:54.562374 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:54.562374 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:54.562374 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:54.562374 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:54.562374 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:54.562374 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:11:54.563186 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:11:54.563186 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:54.563186 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:54.567379 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 238/474] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:11:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:54.568253 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d4bf61-1507-49f0-b673-4c0e42607de1 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:54.669642 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:54.693095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e49870-b775-45a1-bd4f-624a249cb099 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:54.698620 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d4bf61-1507-49f0-b673-4c0e42607de1 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:54.725774 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1886b229-47b6-4ac9-a9f6-5a1e9688176a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:54.726400 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1886b229-47b6-4ac9-a9f6-5a1e9688176a None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:54.726686 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1886b229-47b6-4ac9-a9f6-5a1e9688176a None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:54.726951 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1886b229-47b6-4ac9-a9f6-5a1e9688176a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:54.733955 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1886b229-47b6-4ac9-a9f6-5a1e9688176a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:54.735302 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1886b229-47b6-4ac9-a9f6-5a1e9688176a None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:54.741415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:54.746012 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e167fff-8b72-4766-92f2-be1b048326f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:54.746784 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e167fff-8b72-4766-92f2-be1b048326f5 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:54.747095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-1886b229-47b6-4ac9-a9f6-5a1e9688176a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:11:54.747095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:54.747095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:54.747095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:54.747095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:54.747095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:54.747095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:54.747095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:54.747095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:54.747095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:54.747095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:54.747095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:54.747095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 27 01:11:54.747095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 27 01:11:54.747095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 27 01:11:54.748117 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 27 01:11:54.748117 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:54.748117 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:54.748117 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:54.748117 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:54.748117 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:54.748117 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:54.748117 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:54.748117 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:54.748117 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:54.748117 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:54.748117 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:54.748117 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:54.748117 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:54.748117 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:54.748117 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:54.748117 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:54.749027 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 27 01:11:54.749027 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 27 01:11:54.749027 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 27 01:11:54.749027 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 27 01:11:54.749027 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 27 01:11:54.749027 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 27 01:11:54.749027 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:11:54.749027 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:54.749027 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 237/475] 10.222.0.42 () {56 vars in 1111 bytes} [Thu Oct 27 01:11:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:54.749510 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1886b229-47b6-4ac9-a9f6-5a1e9688176a None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:54.754872 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e167fff-8b72-4766-92f2-be1b048326f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:54.754872 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e167fff-8b72-4766-92f2-be1b048326f5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:54.851583 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e167fff-8b72-4766-92f2-be1b048326f5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:54.853615 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e167fff-8b72-4766-92f2-be1b048326f5 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:54.874330 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1886b229-47b6-4ac9-a9f6-5a1e9688176a None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:54.897195 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:54.928147 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c140763a-9894-429f-9bd8-67ae63c9bd65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:54.928917 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c140763a-9894-429f-9bd8-67ae63c9bd65 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:54.929326 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c140763a-9894-429f-9bd8-67ae63c9bd65 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:54.929698 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c140763a-9894-429f-9bd8-67ae63c9bd65 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:54.930472 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c140763a-9894-429f-9bd8-67ae63c9bd65 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:54.930877 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c140763a-9894-429f-9bd8-67ae63c9bd65 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:54.933577 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c140763a-9894-429f-9bd8-67ae63c9bd65 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:54.949549 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 238/476] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:11:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:54.949549 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c140763a-9894-429f-9bd8-67ae63c9bd65 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:54.976003 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 239/477] 10.222.0.42 () {58 vars in 1282 bytes} [Thu Oct 27 01:11:54 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 408 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 27 01:11:54.977219 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e167fff-8b72-4766-92f2-be1b048326f5 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:55.049436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:55.082429 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c140763a-9894-429f-9bd8-67ae63c9bd65 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:55.084593 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e167fff-8b72-4766-92f2-be1b048326f5 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:55.123173 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58bfd3b4-3f92-4c25-9e2b-b67bc1425995 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:55.123802 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bfd3b4-3f92-4c25-9e2b-b67bc1425995 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:55.124074 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bfd3b4-3f92-4c25-9e2b-b67bc1425995 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:55.124340 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bfd3b4-3f92-4c25-9e2b-b67bc1425995 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:55.130694 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed546e6-4cb8-4c0d-82cc-dffec5b875ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:55.131557 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed546e6-4cb8-4c0d-82cc-dffec5b875ae None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:55.132357 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed546e6-4cb8-4c0d-82cc-dffec5b875ae None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:55.132814 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58bfd3b4-3f92-4c25-9e2b-b67bc1425995 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:55.133025 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed546e6-4cb8-4c0d-82cc-dffec5b875ae None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:55.133981 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ed546e6-4cb8-4c0d-82cc-dffec5b875ae None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:55.134302 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58bfd3b4-3f92-4c25-9e2b-b67bc1425995 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:55.134876 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ed546e6-4cb8-4c0d-82cc-dffec5b875ae None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:55.137227 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ed546e6-4cb8-4c0d-82cc-dffec5b875ae None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:55.146503 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-58bfd3b4-3f92-4c25-9e2b-b67bc1425995 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:11:55.146503 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:55.146503 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:55.146503 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:55.146503 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:55.146503 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:55.146503 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:55.146503 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:55.146503 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:55.146503 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:55.146503 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:55.146503 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:55.146503 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 27 01:11:55.146503 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 27 01:11:55.146503 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 27 01:11:55.147366 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 27 01:11:55.147366 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:55.147366 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:55.147366 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:55.147366 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:55.147366 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:55.147366 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:55.147366 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:55.147366 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:55.147366 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:55.147366 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:55.147366 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:55.147366 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:55.147366 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:55.147366 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:55.147366 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:55.147366 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:55.148206 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 27 01:11:55.148206 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 27 01:11:55.148206 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 27 01:11:55.148206 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 27 01:11:55.148206 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 27 01:11:55.148206 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 27 01:11:55.148206 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:11:55.148206 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:55.152787 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 240/478] 10.222.0.42 () {56 vars in 1111 bytes} [Thu Oct 27 01:11:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:55.153722 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58bfd3b4-3f92-4c25-9e2b-b67bc1425995 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:55.154167 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 239/479] 10.222.0.42 () {56 vars in 1148 bytes} [Thu Oct 27 01:11:54 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:55.155235 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed546e6-4cb8-4c0d-82cc-dffec5b875ae None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:55.217428 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:55.261695 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:55.272589 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58bfd3b4-3f92-4c25-9e2b-b67bc1425995 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:55.295166 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed546e6-4cb8-4c0d-82cc-dffec5b875ae None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:55.313088 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68b5342-6f2f-4ea8-bd00-61639368d203 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:55.313720 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68b5342-6f2f-4ea8-bd00-61639368d203 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:55.313992 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68b5342-6f2f-4ea8-bd00-61639368d203 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:55.314253 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68b5342-6f2f-4ea8-bd00-61639368d203 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:55.325933 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f68b5342-6f2f-4ea8-bd00-61639368d203 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:55.327355 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f68b5342-6f2f-4ea8-bd00-61639368d203 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:55.342166 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a88494c-4b39-4994-b8aa-620c7ed6f9c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:55.343052 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a88494c-4b39-4994-b8aa-620c7ed6f9c4 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:55.343528 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a88494c-4b39-4994-b8aa-620c7ed6f9c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:55.343644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-f68b5342-6f2f-4ea8-bd00-61639368d203 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:55.343644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:55.343644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:55.343644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:55.343644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:55.343644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:55.343644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:55.343644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:55.343644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:55.343644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:55.343644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:55.343644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:55.343644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:11:55.343644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:11:55.343644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:11:55.343644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:11:55.344465 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:55.344465 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:55.344465 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:55.344465 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:55.344465 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:55.344465 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:55.344465 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:55.344465 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:55.344465 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:55.344465 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:55.344465 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:55.344465 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:55.344465 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:55.344465 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:55.344465 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:55.344465 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:55.344465 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:11:55.345937 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:11:55.345937 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:55.345937 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:55.345937 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 241/480] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:11:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:55.345937 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68b5342-6f2f-4ea8-bd00-61639368d203 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:55.346725 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a88494c-4b39-4994-b8aa-620c7ed6f9c4 None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:55.347918 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a88494c-4b39-4994-b8aa-620c7ed6f9c4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:55.348423 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a88494c-4b39-4994-b8aa-620c7ed6f9c4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:55.350301 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a88494c-4b39-4994-b8aa-620c7ed6f9c4 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:55.370605 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 240/481] 10.222.0.42 () {56 vars in 1148 bytes} [Thu Oct 27 01:11:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:55.371882 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a88494c-4b39-4994-b8aa-620c7ed6f9c4 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:55.472437 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:55.480652 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a88494c-4b39-4994-b8aa-620c7ed6f9c4 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:55.482105 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68b5342-6f2f-4ea8-bd00-61639368d203 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:55.517464 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b279bd-1ee0-4e02-bfb9-3506bc303ec9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:55.518385 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b279bd-1ee0-4e02-bfb9-3506bc303ec9 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:55.518835 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b279bd-1ee0-4e02-bfb9-3506bc303ec9 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:55.519326 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b279bd-1ee0-4e02-bfb9-3506bc303ec9 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:55.520684 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b279bd-1ee0-4e02-bfb9-3506bc303ec9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:55.521204 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b279bd-1ee0-4e02-bfb9-3506bc303ec9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:55.522960 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b279bd-1ee0-4e02-bfb9-3506bc303ec9 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:55.526092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12524ca7-bd3a-43e5-b606-67b2f5e2369c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:55.526878 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12524ca7-bd3a-43e5-b606-67b2f5e2369c None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:55.527304 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12524ca7-bd3a-43e5-b606-67b2f5e2369c None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:55.527715 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12524ca7-bd3a-43e5-b606-67b2f5e2369c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:55.528570 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12524ca7-bd3a-43e5-b606-67b2f5e2369c None admin] RBAC: Adding query filter params (role.id=1b8f8dd9e8584053b048032db63254a4, scope.project.id=23aad848d47c4d27ad4c138ff869e40f, user.id=2880dbb00eec471285d242e8183dd8d0) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:55.529036 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12524ca7-bd3a-43e5-b606-67b2f5e2369c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:55.531123 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12524ca7-bd3a-43e5-b606-67b2f5e2369c None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:55.536709 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 241/482] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:11:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:55.537817 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b279bd-1ee0-4e02-bfb9-3506bc303ec9 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:55.540009 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 242/483] 10.222.0.42 () {56 vars in 1374 bytes} [Thu Oct 27 01:11:55 2022] GET /identity/v3/role_assignments?user.id=2880dbb00eec471285d242e8183dd8d0&scope.project.id=23aad848d47c4d27ad4c138ff869e40f&role.id=1b8f8dd9e8584053b048032db63254a4 => generated 608 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:55.541702 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12524ca7-bd3a-43e5-b606-67b2f5e2369c None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:55.554661 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:55.670626 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12524ca7-bd3a-43e5-b606-67b2f5e2369c None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:55.716036 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4b8ed3-773c-4864-8d7d-cae41cf88613 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:55.717396 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4b8ed3-773c-4864-8d7d-cae41cf88613 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:55.717839 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4b8ed3-773c-4864-8d7d-cae41cf88613 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:55.718244 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4b8ed3-773c-4864-8d7d-cae41cf88613 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:55.728369 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4b8ed3-773c-4864-8d7d-cae41cf88613 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:55.730945 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4b8ed3-773c-4864-8d7d-cae41cf88613 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:55.744983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-fd4b8ed3-773c-4864-8d7d-cae41cf88613 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:11:55.744983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:55.744983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:55.744983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:55.744983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:55.744983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:55.744983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:55.744983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:55.744983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:55.744983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:55.744983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:55.744983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:55.744983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 27 01:11:55.744983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 27 01:11:55.744983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 27 01:11:55.746139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 27 01:11:55.746139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:55.746139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:55.746139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:55.746139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:55.746139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:55.746139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:55.746139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:55.746139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:55.746139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:55.746139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:55.746139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:55.746139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:55.746139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:55.746139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:55.746139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:55.746139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:55.747084 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 27 01:11:55.747084 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 27 01:11:55.747084 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 27 01:11:55.747084 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 27 01:11:55.747084 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 27 01:11:55.747084 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 27 01:11:55.747084 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:11:55.747084 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:55.749605 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 243/484] 10.222.0.42 () {56 vars in 1111 bytes} [Thu Oct 27 01:11:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 27 01:11:55.756626 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4b8ed3-773c-4864-8d7d-cae41cf88613 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:55.776135 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b279bd-1ee0-4e02-bfb9-3506bc303ec9 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:55.832696 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5715f5d4-41ff-4051-9acc-064179989233 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:55.833436 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5715f5d4-41ff-4051-9acc-064179989233 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:55.834582 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5715f5d4-41ff-4051-9acc-064179989233 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:55.836878 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5715f5d4-41ff-4051-9acc-064179989233 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:55.838455 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5715f5d4-41ff-4051-9acc-064179989233 None admin] RBAC: Adding query filter params (role.id=1b8f8dd9e8584053b048032db63254a4, scope.project.id=23aad848d47c4d27ad4c138ff869e40f, user.id=d35a3d40a1924064859562a72bab0f6d) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:55.838455 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5715f5d4-41ff-4051-9acc-064179989233 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:55.839466 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5715f5d4-41ff-4051-9acc-064179989233 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:55.847395 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 242/485] 10.222.0.42 () {56 vars in 1374 bytes} [Thu Oct 27 01:11:55 2022] GET /identity/v3/role_assignments?user.id=d35a3d40a1924064859562a72bab0f6d&scope.project.id=23aad848d47c4d27ad4c138ff869e40f&role.id=1b8f8dd9e8584053b048032db63254a4 => generated 608 bytes in 310 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:55.901455 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:55.902933 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d128dcb-8bf7-4d74-970a-23ea783fe4c0 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:55.903352 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d128dcb-8bf7-4d74-970a-23ea783fe4c0 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:55.903682 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d128dcb-8bf7-4d74-970a-23ea783fe4c0 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:55.906080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 243/486] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:55.912083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef14ed2-d49f-473d-b73e-6e61fda35b48 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:55.912395 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef14ed2-d49f-473d-b73e-6e61fda35b48 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:55.912628 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef14ed2-d49f-473d-b73e-6e61fda35b48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:55.935825 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4b8ed3-773c-4864-8d7d-cae41cf88613 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:55.941503 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-1ef14ed2-d49f-473d-b73e-6e61fda35b48 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:55.942733 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-1ef14ed2-d49f-473d-b73e-6e61fda35b48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:55.965494 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:55.978369 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12334848-64aa-4643-867f-37eae1d2f230 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:55.979319 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12334848-64aa-4643-867f-37eae1d2f230 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:55.979779 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12334848-64aa-4643-867f-37eae1d2f230 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:55.980322 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12334848-64aa-4643-867f-37eae1d2f230 None admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:55.981247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12334848-64aa-4643-867f-37eae1d2f230 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:55.981776 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12334848-64aa-4643-867f-37eae1d2f230 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:55.983428 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12334848-64aa-4643-867f-37eae1d2f230 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:56.012032 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 244/487] 10.222.0.42 () {56 vars in 1148 bytes} [Thu Oct 27 01:11:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:56.013413 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12334848-64aa-4643-867f-37eae1d2f230 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:56.037437 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 244/488] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 27 01:11:56.130412 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12334848-64aa-4643-867f-37eae1d2f230 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:56.164956 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:56.173672 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27fa224f-bb3d-424c-8eb2-ecc0a0dbf644 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:56.174280 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fa224f-bb3d-424c-8eb2-ecc0a0dbf644 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:56.174553 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fa224f-bb3d-424c-8eb2-ecc0a0dbf644 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:56.174821 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fa224f-bb3d-424c-8eb2-ecc0a0dbf644 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:56.176044 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27fa224f-bb3d-424c-8eb2-ecc0a0dbf644 None admin] RBAC: Adding query filter params (role.id=1b8f8dd9e8584053b048032db63254a4, scope.project.id=23aad848d47c4d27ad4c138ff869e40f, user.id=4072417990c94b0488871332d28224ba) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:56.176489 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27fa224f-bb3d-424c-8eb2-ecc0a0dbf644 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:56.177942 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27fa224f-bb3d-424c-8eb2-ecc0a0dbf644 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:56.184854 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 245/489] 10.222.0.42 () {56 vars in 1374 bytes} [Thu Oct 27 01:11:56 2022] GET /identity/v3/role_assignments?user.id=4072417990c94b0488871332d28224ba&scope.project.id=23aad848d47c4d27ad4c138ff869e40f&role.id=1b8f8dd9e8584053b048032db63254a4 => generated 608 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:56.474049 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef14ed2-d49f-473d-b73e-6e61fda35b48 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:56.580483 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef14ed2-d49f-473d-b73e-6e61fda35b48 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:56.624837 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3a9771-dd9f-40be-babb-d01b2faaa7dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:56.625516 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3a9771-dd9f-40be-babb-d01b2faaa7dd None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:56.625794 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3a9771-dd9f-40be-babb-d01b2faaa7dd None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:56.626063 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3a9771-dd9f-40be-babb-d01b2faaa7dd None admin] PATH_INFO: `/v3/roles/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:56.639157 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3a9771-dd9f-40be-babb-d01b2faaa7dd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:56.646257 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-3f3a9771-dd9f-40be-babb-d01b2faaa7dd None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 27 01:11:56.646257 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:56.646257 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:56.646257 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:56.646257 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:56.646257 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:56.646257 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:56.646257 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:56.646257 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:56.646257 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:56.646257 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:56.646257 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:56.646257 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:56.646257 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:56.646257 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:56.646257 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:56.647152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:56.647152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:56.647152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:56.647152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:56.647152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:56.647152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:56.647152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:56.647152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:56.647152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:56.647152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:56.647152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:56.647152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:56.647152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:56.647152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:56.647152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:56.647152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:56.647946 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:56.647946 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:56.647946 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:56.647946 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:56.647946 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:56.647946 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:56.647946 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:56.647946 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:56.647946 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:56.647946 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:56.647946 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 27 01:11:56.647946 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:56.647946 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 245/490] 10.222.0.42 () {56 vars in 1102 bytes} [Thu Oct 27 01:11:56 2022] GET /identity/v3/roles/service => generated 85 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:56.654169 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27fa224f-bb3d-424c-8eb2-ecc0a0dbf644 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:56.722147 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15da52e9-229f-41d4-a1e2-61fdf25c7c25 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:56.722609 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15da52e9-229f-41d4-a1e2-61fdf25c7c25 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:56.722799 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15da52e9-229f-41d4-a1e2-61fdf25c7c25 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:56.724131 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 246/491] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 27 01:11:56.729377 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ce293b-a314-4410-8367-64ae03a0f5d6 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:56.729656 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ce293b-a314-4410-8367-64ae03a0f5d6 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:56.729923 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ce293b-a314-4410-8367-64ae03a0f5d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:56.751138 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-62ce293b-a314-4410-8367-64ae03a0f5d6 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:56.752753 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-62ce293b-a314-4410-8367-64ae03a0f5d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:56.761396 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:56.857175 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27fa224f-bb3d-424c-8eb2-ecc0a0dbf644 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:56.903099 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 247/492] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 27 01:11:56.905103 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6a3805-da30-4cb0-bcad-09276350fc57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:56.905797 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6a3805-da30-4cb0-bcad-09276350fc57 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:56.906081 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6a3805-da30-4cb0-bcad-09276350fc57 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:56.906398 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6a3805-da30-4cb0-bcad-09276350fc57 None admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:56.907122 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b6a3805-da30-4cb0-bcad-09276350fc57 None admin] RBAC: Adding query filter params (name=service) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:56.907424 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b6a3805-da30-4cb0-bcad-09276350fc57 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:56.908790 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b6a3805-da30-4cb0-bcad-09276350fc57 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:56.918114 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 246/493] 10.222.0.42 () {56 vars in 1103 bytes} [Thu Oct 27 01:11:56 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:56.926672 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ce293b-a314-4410-8367-64ae03a0f5d6 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:57.034189 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ce293b-a314-4410-8367-64ae03a0f5d6 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:57.070710 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7907b2-e4c0-417f-9b0f-657d25e67c40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:57.071344 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7907b2-e4c0-417f-9b0f-657d25e67c40 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:57.071625 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7907b2-e4c0-417f-9b0f-657d25e67c40 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:57.071902 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7907b2-e4c0-417f-9b0f-657d25e67c40 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:57.080839 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:57.083370 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af7907b2-e4c0-417f-9b0f-657d25e67c40 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:57.084882 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af7907b2-e4c0-417f-9b0f-657d25e67c40 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:57.100683 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-af7907b2-e4c0-417f-9b0f-657d25e67c40 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:57.100683 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:57.100683 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:57.100683 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:57.100683 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:57.100683 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:57.100683 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:57.100683 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:57.100683 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:57.100683 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:57.100683 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:57.100683 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:57.100683 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:11:57.100683 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:11:57.100683 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:11:57.100683 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:11:57.102663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:57.102663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:57.102663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:57.102663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:57.102663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:57.102663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:57.102663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:57.102663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:57.102663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:57.102663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:57.102663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:57.102663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:57.102663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:57.102663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:57.102663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:57.102663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:57.102663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:11:57.103865 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:11:57.103865 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:57.103865 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:57.103865 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 248/494] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:11:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:57.107520 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6a3805-da30-4cb0-bcad-09276350fc57 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:57.179249 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:57.241319 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6a3805-da30-4cb0-bcad-09276350fc57 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:57.269648 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7907b2-e4c0-417f-9b0f-657d25e67c40 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:57.279506 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5c6132-0416-437c-afef-a1cbaace8eab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:57.280121 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5c6132-0416-437c-afef-a1cbaace8eab None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:57.280398 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5c6132-0416-437c-afef-a1cbaace8eab None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:57.280671 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5c6132-0416-437c-afef-a1cbaace8eab None admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:57.281349 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5c6132-0416-437c-afef-a1cbaace8eab None admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:57.281654 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5c6132-0416-437c-afef-a1cbaace8eab None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:57.283034 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5c6132-0416-437c-afef-a1cbaace8eab None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:57.296029 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 247/495] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:11:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:57.301299 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5c6132-0416-437c-afef-a1cbaace8eab None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:57.412684 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7907b2-e4c0-417f-9b0f-657d25e67c40 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:57.448680 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd48d65-3b6b-48b6-bac8-6e9031509117 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:57.449345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd48d65-3b6b-48b6-bac8-6e9031509117 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:57.449624 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd48d65-3b6b-48b6-bac8-6e9031509117 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:57.449894 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd48d65-3b6b-48b6-bac8-6e9031509117 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:57.461245 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfd48d65-3b6b-48b6-bac8-6e9031509117 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:11:57.467092 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-bfd48d65-3b6b-48b6-bac8-6e9031509117 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:11:57.467092 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:57.467092 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:57.467092 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:57.467092 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:57.467092 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:57.467092 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:57.467092 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:57.467092 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:57.467092 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:57.467092 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:57.467092 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:57.467092 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:11:57.467092 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:11:57.467092 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:11:57.467092 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:11:57.468008 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:11:57.468008 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:11:57.468008 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:11:57.468008 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:11:57.468008 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:57.468008 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:57.468008 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:57.468008 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:57.468008 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:57.468008 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:57.468008 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:57.468008 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:57.468008 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:57.468008 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:57.468008 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:57.468008 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:57.469367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:57.469367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:57.469367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:57.469367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:57.469367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:11:57.469367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:11:57.469367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:11:57.469367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:11:57.469367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:11:57.469367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:11:57.469367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:11:57.469367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:57.469367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 249/496] 10.222.0.42 () {56 vars in 1096 bytes} [Thu Oct 27 01:11:57 2022] GET /identity/v3/roles/admin => generated 83 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:57.474469 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd48d65-3b6b-48b6-bac8-6e9031509117 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:57.541448 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:57.593402 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5c6132-0416-437c-afef-a1cbaace8eab None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:57.625407 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:57.633847 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd48d65-3b6b-48b6-bac8-6e9031509117 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:57.667659 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b464eee7-bcc5-44be-9735-1cd635cd6f39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:57.668351 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b464eee7-bcc5-44be-9735-1cd635cd6f39 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:57.668633 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b464eee7-bcc5-44be-9735-1cd635cd6f39 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:57.668905 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b464eee7-bcc5-44be-9735-1cd635cd6f39 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:57.682713 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b464eee7-bcc5-44be-9735-1cd635cd6f39 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:57.684167 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b464eee7-bcc5-44be-9735-1cd635cd6f39 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:57.701746 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4673a2-ea6f-4fae-ae79-9a6f0e96b401 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:57.702359 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4673a2-ea6f-4fae-ae79-9a6f0e96b401 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:57.702635 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4673a2-ea6f-4fae-ae79-9a6f0e96b401 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:57.702904 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4673a2-ea6f-4fae-ae79-9a6f0e96b401 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:57.703570 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4673a2-ea6f-4fae-ae79-9a6f0e96b401 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:57.703873 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4673a2-ea6f-4fae-ae79-9a6f0e96b401 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:57.705305 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4673a2-ea6f-4fae-ae79-9a6f0e96b401 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:57.712292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-b464eee7-bcc5-44be-9735-1cd635cd6f39 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 27 01:11:57.712292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:57.712292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:57.712292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:57.712292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:57.712292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:57.712292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:57.712292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:57.712292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:57.712292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:57.712292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:57.712292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:57.712292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 27 01:11:57.712292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 27 01:11:57.712292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 27 01:11:57.712292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 27 01:11:57.713242 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:57.713242 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:57.713242 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 27 01:11:57.713242 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:11:57.713242 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 27 01:11:57.713242 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:11:57.713242 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:57.713242 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:57.713242 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:57.713242 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:57.713242 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:57.713242 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:57.713242 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:57.713242 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:57.713242 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:57.713242 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:57.713242 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:57.713242 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:57.715049 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:57.715049 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:57.715049 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 27 01:11:57.715049 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 27 01:11:57.715049 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 27 01:11:57.715049 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 27 01:11:57.715049 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 27 01:11:57.715049 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 27 01:11:57.715049 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 27 01:11:57.715049 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:57.715049 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 248/497] 10.222.0.42 () {56 vars in 1096 bytes} [Thu Oct 27 01:11:57 2022] GET /identity/v3/users/swift => generated 83 bytes in 413 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:57.724086 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b464eee7-bcc5-44be-9735-1cd635cd6f39 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:57.731075 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 250/498] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:11:57 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:11:57.738152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4673a2-ea6f-4fae-ae79-9a6f0e96b401 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:57.906136 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b464eee7-bcc5-44be-9735-1cd635cd6f39 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:57.935313 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4673a2-ea6f-4fae-ae79-9a6f0e96b401 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:57.953092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c5e3a6-f3d7-4116-9d75-5ce3ed7ea771 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:57.953710 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c5e3a6-f3d7-4116-9d75-5ce3ed7ea771 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:57.953989 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c5e3a6-f3d7-4116-9d75-5ce3ed7ea771 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:57.954261 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c5e3a6-f3d7-4116-9d75-5ce3ed7ea771 None admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:57.954993 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c5e3a6-f3d7-4116-9d75-5ce3ed7ea771 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:57.955298 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c5e3a6-f3d7-4116-9d75-5ce3ed7ea771 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:57.957460 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c5e3a6-f3d7-4116-9d75-5ce3ed7ea771 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:57.968968 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:57.991282 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 249/499] 10.222.0.42 () {56 vars in 1135 bytes} [Thu Oct 27 01:11:57 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:57.997490 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c5e3a6-f3d7-4116-9d75-5ce3ed7ea771 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:58.006061 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b443d4-ddc5-49bf-8715-65286d7ebbcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:58.006813 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b443d4-ddc5-49bf-8715-65286d7ebbcb None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:58.007093 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b443d4-ddc5-49bf-8715-65286d7ebbcb None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:58.007367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b443d4-ddc5-49bf-8715-65286d7ebbcb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:58.019977 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67b443d4-ddc5-49bf-8715-65286d7ebbcb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:58.021476 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67b443d4-ddc5-49bf-8715-65286d7ebbcb None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:58.037216 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-67b443d4-ddc5-49bf-8715-65286d7ebbcb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:58.037216 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:58.037216 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:58.037216 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:58.037216 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:58.037216 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:58.037216 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:58.037216 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:58.037216 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:58.037216 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:58.037216 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:58.037216 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:58.037216 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:11:58.037216 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:11:58.037216 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:11:58.037216 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:11:58.038164 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:58.038164 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:58.038164 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:58.038164 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:58.038164 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:58.038164 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:58.038164 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:58.038164 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:58.038164 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:58.038164 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:58.038164 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:58.038164 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:58.038164 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:58.038164 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:58.038164 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:58.038164 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:58.038164 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:11:58.040110 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:11:58.040110 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:58.040110 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:58.040110 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 251/500] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:11:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 303 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 27 01:11:58.044732 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b443d4-ddc5-49bf-8715-65286d7ebbcb None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:58.070884 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:58.194874 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c5e3a6-f3d7-4116-9d75-5ce3ed7ea771 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:58.235659 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b443d4-ddc5-49bf-8715-65286d7ebbcb None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:58.251010 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60fd6e8-cf52-4c84-99c7-c2c29842cded None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:58.252097 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60fd6e8-cf52-4c84-99c7-c2c29842cded None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:58.252576 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60fd6e8-cf52-4c84-99c7-c2c29842cded None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:58.253096 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60fd6e8-cf52-4c84-99c7-c2c29842cded None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:58.321895 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c92d0c6-e8d7-4813-bd00-bcfa36b604d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:58.322514 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c92d0c6-e8d7-4813-bd00-bcfa36b604d8 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:58.322788 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c92d0c6-e8d7-4813-bd00-bcfa36b604d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:58.323055 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c92d0c6-e8d7-4813-bd00-bcfa36b604d8 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:58.323928 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c92d0c6-e8d7-4813-bd00-bcfa36b604d8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:58.324038 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c92d0c6-e8d7-4813-bd00-bcfa36b604d8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:58.325485 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c92d0c6-e8d7-4813-bd00-bcfa36b604d8 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:58.334862 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60fd6e8-cf52-4c84-99c7-c2c29842cded None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:58.336661 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60fd6e8-cf52-4c84-99c7-c2c29842cded None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:58.351961 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 252/501] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:11:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 307 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:58.356995 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c92d0c6-e8d7-4813-bd00-bcfa36b604d8 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:58.371530 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:58.411309 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:58.476164 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 250/502] 10.222.0.42 () {58 vars in 1282 bytes} [Thu Oct 27 01:11:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 479 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 27 01:11:58.477407 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7490c0a-8bbf-47de-b3cf-5c32337fd299 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:58.477687 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7490c0a-8bbf-47de-b3cf-5c32337fd299 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:58.477957 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7490c0a-8bbf-47de-b3cf-5c32337fd299 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:58.479168 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 251/503] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:58.480151 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4830573e-9634-42cb-828b-0a919a041490 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:58.480421 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4830573e-9634-42cb-828b-0a919a041490 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:58.480758 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4830573e-9634-42cb-828b-0a919a041490 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:58.482169 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 252/504] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:58 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:58.482997 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4830573e-9634-42cb-828b-0a919a041490 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:58.497856 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c92d0c6-e8d7-4813-bd00-bcfa36b604d8 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:58.536340 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a3ce47-02df-4976-b3f2-426450de4eed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:58.536967 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a3ce47-02df-4976-b3f2-426450de4eed None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:58.537277 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a3ce47-02df-4976-b3f2-426450de4eed None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:58.537555 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a3ce47-02df-4976-b3f2-426450de4eed None admin] PATH_INFO: `/v3/users/nova` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:58.544968 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a3ce47-02df-4976-b3f2-426450de4eed None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:58.546435 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a3ce47-02df-4976-b3f2-426450de4eed None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:58.559235 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-c3a3ce47-02df-4976-b3f2-426450de4eed None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 27 01:11:58.559235 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:58.559235 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:58.559235 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:58.559235 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:58.559235 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:58.559235 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:58.559235 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:58.559235 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:58.559235 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:58.559235 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:58.559235 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:58.559235 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 27 01:11:58.559235 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 27 01:11:58.559235 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 27 01:11:58.559235 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 27 01:11:58.560395 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:58.560395 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:58.560395 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 27 01:11:58.560395 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:11:58.560395 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 27 01:11:58.560395 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:11:58.560395 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:58.560395 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:58.560395 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:58.560395 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:58.560395 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:58.560395 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:58.560395 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:58.560395 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:58.560395 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:58.560395 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:58.560395 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:58.560395 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:58.561661 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:58.561661 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:58.561661 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 27 01:11:58.561661 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 27 01:11:58.561661 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 27 01:11:58.561661 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 27 01:11:58.561661 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 27 01:11:58.561661 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 27 01:11:58.561661 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 27 01:11:58.561661 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:58.565087 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 253/505] 10.222.0.42 () {56 vars in 1093 bytes} [Thu Oct 27 01:11:58 2022] GET /identity/v3/users/nova => generated 82 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:58.566377 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274ae403-1e4d-432f-a902-c14991ab8859 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:58.566659 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274ae403-1e4d-432f-a902-c14991ab8859 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:58.566932 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274ae403-1e4d-432f-a902-c14991ab8859 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:58.585994 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-274ae403-1e4d-432f-a902-c14991ab8859 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:58.587183 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-274ae403-1e4d-432f-a902-c14991ab8859 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:58.602113 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4830573e-9634-42cb-828b-0a919a041490 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:58.636135 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:58.641968 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f1855d5-bb0e-4c29-9a9f-e38c57b0a446 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:58.642592 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1855d5-bb0e-4c29-9a9f-e38c57b0a446 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:58.642933 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1855d5-bb0e-4c29-9a9f-e38c57b0a446 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:58.643197 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1855d5-bb0e-4c29-9a9f-e38c57b0a446 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:58.654472 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f1855d5-bb0e-4c29-9a9f-e38c57b0a446 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:58.655919 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f1855d5-bb0e-4c29-9a9f-e38c57b0a446 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:58.659268 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 254/506] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:11:58.670544 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7ef3c3-9d72-4ba4-9018-c2ebf6c88040 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:58.671085 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7ef3c3-9d72-4ba4-9018-c2ebf6c88040 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:58.671622 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7ef3c3-9d72-4ba4-9018-c2ebf6c88040 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:58.672048 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-3f1855d5-bb0e-4c29-9a9f-e38c57b0a446 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:58.672048 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:58.672048 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:58.672048 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:58.672048 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:58.672048 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:58.672048 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:58.672048 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:58.672048 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:58.672048 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:58.672048 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:58.672048 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:58.672048 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:11:58.672048 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:11:58.672048 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:11:58.672048 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:11:58.672953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:58.672953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:58.672953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:58.672953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:58.672953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:58.672953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:58.672953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:58.672953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:58.672953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:58.672953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:58.672953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:58.672953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:58.672953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:58.672953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:58.672953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:58.672953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:58.672953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:11:58.674549 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:11:58.674549 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:58.674549 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:58.674549 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 253/507] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:11:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:58.674549 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f1855d5-bb0e-4c29-9a9f-e38c57b0a446 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:58.700763 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-7f7ef3c3-9d72-4ba4-9018-c2ebf6c88040 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:58.703047 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-7f7ef3c3-9d72-4ba4-9018-c2ebf6c88040 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:58.718843 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:58.788110 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f1855d5-bb0e-4c29-9a9f-e38c57b0a446 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:58.793684 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 255/508] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:11:58.794859 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f7ef3c3-9d72-4ba4-9018-c2ebf6c88040 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:58.835333 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2efb05bf-e255-4fa4-ba2c-1c0a19abdd12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:58.835917 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2efb05bf-e255-4fa4-ba2c-1c0a19abdd12 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:58.836194 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2efb05bf-e255-4fa4-ba2c-1c0a19abdd12 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:58.836459 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2efb05bf-e255-4fa4-ba2c-1c0a19abdd12 None admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:58.837259 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2efb05bf-e255-4fa4-ba2c-1c0a19abdd12 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:58.837497 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2efb05bf-e255-4fa4-ba2c-1c0a19abdd12 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:58.838981 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2efb05bf-e255-4fa4-ba2c-1c0a19abdd12 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:58.858065 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 254/509] 10.222.0.42 () {56 vars in 1133 bytes} [Thu Oct 27 01:11:58 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:58.865281 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2efb05bf-e255-4fa4-ba2c-1c0a19abdd12 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:58.918058 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:58.937473 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f7ef3c3-9d72-4ba4-9018-c2ebf6c88040 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:58.979175 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc79060-ee00-4447-adf6-7603a99c0dba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:58.980117 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc79060-ee00-4447-adf6-7603a99c0dba None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:58.981539 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc79060-ee00-4447-adf6-7603a99c0dba None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:58.981994 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc79060-ee00-4447-adf6-7603a99c0dba None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:58.982813 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dc79060-ee00-4447-adf6-7603a99c0dba None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:58.983287 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dc79060-ee00-4447-adf6-7603a99c0dba None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:58.984843 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dc79060-ee00-4447-adf6-7603a99c0dba None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:58.998123 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 256/510] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:11:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:59.000292 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc146bee-7ed1-4457-871a-76d89f6305bc None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:59.000920 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc146bee-7ed1-4457-871a-76d89f6305bc None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:59.001394 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc146bee-7ed1-4457-871a-76d89f6305bc None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:59.003253 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 257/511] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:11:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:11:59.004383 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc146bee-7ed1-4457-871a-76d89f6305bc None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:59.014996 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:59.034844 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2efb05bf-e255-4fa4-ba2c-1c0a19abdd12 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:59.073210 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f20478a-328f-4814-ac6d-ae7400336393 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:59.073760 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f20478a-328f-4814-ac6d-ae7400336393 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:59.074039 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f20478a-328f-4814-ac6d-ae7400336393 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:59.074307 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f20478a-328f-4814-ac6d-ae7400336393 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:59.087052 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f20478a-328f-4814-ac6d-ae7400336393 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:59.089363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f20478a-328f-4814-ac6d-ae7400336393 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:59.108074 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-0f20478a-328f-4814-ac6d-ae7400336393 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:59.108074 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:59.108074 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:59.108074 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:59.108074 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:59.108074 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:59.108074 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:59.108074 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:59.108074 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:59.108074 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:59.108074 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:59.108074 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:59.108074 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:11:59.108074 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:11:59.108074 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:11:59.108074 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:11:59.109037 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:59.109037 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:59.109037 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:59.109037 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:59.109037 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:59.109037 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:59.109037 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:59.109037 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:59.109037 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:59.109037 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:59.109037 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:59.109037 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:59.109037 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:59.109037 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:59.109037 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:59.109037 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:59.109037 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:11:59.115312 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:11:59.115312 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:59.115312 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:59.115312 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 255/512] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:11:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:59.115312 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c357e474-856b-4727-b06a-46b161ab89be None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:59.115312 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c357e474-856b-4727-b06a-46b161ab89be None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:59.115312 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c357e474-856b-4727-b06a-46b161ab89be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:59.118009 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc146bee-7ed1-4457-871a-76d89f6305bc None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:59.132451 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-c357e474-856b-4727-b06a-46b161ab89be None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:11:59.133663 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-c357e474-856b-4727-b06a-46b161ab89be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:11:59.151336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696fd644-7beb-4577-8ab3-fb4156f0c7bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:59.151959 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696fd644-7beb-4577-8ab3-fb4156f0c7bb None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:59.152236 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696fd644-7beb-4577-8ab3-fb4156f0c7bb None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:59.152508 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696fd644-7beb-4577-8ab3-fb4156f0c7bb None admin] PATH_INFO: `/v3/projects/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:59.158963 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696fd644-7beb-4577-8ab3-fb4156f0c7bb None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:59.160980 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696fd644-7beb-4577-8ab3-fb4156f0c7bb None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:59.171818 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-696fd644-7beb-4577-8ab3-fb4156f0c7bb None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:11:59.171818 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:59.171818 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:59.171818 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:59.171818 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:59.171818 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:59.171818 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:59.171818 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:59.171818 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:59.171818 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:59.171818 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:59.171818 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:59.171818 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 27 01:11:59.171818 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 27 01:11:59.171818 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 27 01:11:59.172726 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 27 01:11:59.172726 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:59.172726 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:59.172726 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:59.172726 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:59.172726 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:59.172726 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:59.172726 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:59.172726 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:59.172726 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:59.172726 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:59.172726 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:59.172726 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:59.172726 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:59.172726 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:59.172726 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:59.172726 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:59.173679 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 27 01:11:59.173679 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 27 01:11:59.173679 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 27 01:11:59.173679 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 27 01:11:59.173679 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 27 01:11:59.173679 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 27 01:11:59.173679 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:11:59.173679 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:59.173679 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 258/513] 10.222.0.42 () {56 vars in 1111 bytes} [Thu Oct 27 01:11:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:59.173679 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696fd644-7beb-4577-8ab3-fb4156f0c7bb None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:59.201682 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 256/514] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:11:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:11:59.202514 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c357e474-856b-4727-b06a-46b161ab89be None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:59.228367 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:59.240375 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:59.277471 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696fd644-7beb-4577-8ab3-fb4156f0c7bb None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:59.306727 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c357e474-856b-4727-b06a-46b161ab89be None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:59.313278 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bace78f4-f50a-47b3-a532-669a8ac56fba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:59.313902 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bace78f4-f50a-47b3-a532-669a8ac56fba None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:59.314174 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bace78f4-f50a-47b3-a532-669a8ac56fba None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:59.314439 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bace78f4-f50a-47b3-a532-669a8ac56fba None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:59.326486 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bace78f4-f50a-47b3-a532-669a8ac56fba None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:59.327860 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bace78f4-f50a-47b3-a532-669a8ac56fba None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:59.342330 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-164b6b8f-ed6e-4c02-b3cd-51138c2a2469 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:59.342932 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164b6b8f-ed6e-4c02-b3cd-51138c2a2469 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:59.343187 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-bace78f4-f50a-47b3-a532-669a8ac56fba None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:59.343187 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:59.343187 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:59.343187 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:59.343187 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:59.343187 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:59.343187 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:59.343187 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:59.343187 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:59.343187 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:59.343187 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:59.343187 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:59.343187 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:11:59.343187 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:11:59.343187 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:11:59.343187 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:11:59.344095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:59.344095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:59.344095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:59.344095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:59.344095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:59.344095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:59.344095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:59.344095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:59.344095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:59.344095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:59.344095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:59.344095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:59.344095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:59.344095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:59.344095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:59.344095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:59.344095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:11:59.350457 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:11:59.350457 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:11:59.350457 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:59.350632 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164b6b8f-ed6e-4c02-b3cd-51138c2a2469 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:59.350632 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164b6b8f-ed6e-4c02-b3cd-51138c2a2469 None admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:59.350632 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-164b6b8f-ed6e-4c02-b3cd-51138c2a2469 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:59.350632 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-164b6b8f-ed6e-4c02-b3cd-51138c2a2469 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:59.350900 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 259/515] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:11:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:59.350900 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bace78f4-f50a-47b3-a532-669a8ac56fba None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:59.351022 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-164b6b8f-ed6e-4c02-b3cd-51138c2a2469 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:59.362247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 257/516] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:11:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:59.363245 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-164b6b8f-ed6e-4c02-b3cd-51138c2a2469 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:59.451991 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:59.469046 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-164b6b8f-ed6e-4c02-b3cd-51138c2a2469 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:59.474287 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bace78f4-f50a-47b3-a532-669a8ac56fba None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:59.493288 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:59.503094 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c24dcc4-0911-4820-8552-55410887258f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:59.503755 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c24dcc4-0911-4820-8552-55410887258f None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:59.504054 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c24dcc4-0911-4820-8552-55410887258f None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:59.504437 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c24dcc4-0911-4820-8552-55410887258f None admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:59.505161 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c24dcc4-0911-4820-8552-55410887258f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:59.505464 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c24dcc4-0911-4820-8552-55410887258f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:59.506883 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c24dcc4-0911-4820-8552-55410887258f None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:59.510345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c94cdc-2c23-49a9-834e-a41c22f765be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:59.510995 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c94cdc-2c23-49a9-834e-a41c22f765be None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:59.511281 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c94cdc-2c23-49a9-834e-a41c22f765be None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:59.511553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c94cdc-2c23-49a9-834e-a41c22f765be None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:59.512134 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c94cdc-2c23-49a9-834e-a41c22f765be None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:59.513526 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c94cdc-2c23-49a9-834e-a41c22f765be None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:59.518697 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 258/517] 10.222.0.42 () {56 vars in 1148 bytes} [Thu Oct 27 01:11:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:59.519364 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c24dcc4-0911-4820-8552-55410887258f None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:59.525282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-66c94cdc-2c23-49a9-834e-a41c22f765be None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 27 01:11:59.525282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:59.525282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 27 01:11:59.525282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 27 01:11:59.525282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 27 01:11:59.525282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 27 01:11:59.525282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 27 01:11:59.525282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:11:59.525282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 27 01:11:59.525282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:59.525282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 27 01:11:59.525282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:59.525282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:59.525282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:59.525282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:59.525282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:59.525282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:59.526427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:59.526427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:59.526427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:59.526427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:59.526427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:59.526427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:59.526427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 27 01:11:59.526427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 27 01:11:59.526427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 27 01:11:59.526427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 27 01:11:59.526427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:59.526427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:59.526427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:59.526427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:59.526427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:59.526427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:59.527277 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:59.527277 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:59.527277 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:59.527277 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:59.527277 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:59.527277 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:59.527277 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:59.527277 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:59.527277 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:59.527277 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:59.527277 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 27 01:11:59.527277 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:11:59.527277 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 27 01:11:59.527277 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:59.527277 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 260/518] 10.222.0.42 () {56 vars in 1129 bytes} [Thu Oct 27 01:11:59 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:59.528090 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c94cdc-2c23-49a9-834e-a41c22f765be None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:59.624084 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c24dcc4-0911-4820-8552-55410887258f None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:59.634229 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c94cdc-2c23-49a9-834e-a41c22f765be None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:59.658758 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe1418a6-58d8-4328-a73d-b80acd4342ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:59.659354 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1418a6-58d8-4328-a73d-b80acd4342ce None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:59.659633 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1418a6-58d8-4328-a73d-b80acd4342ce None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:59.659939 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1418a6-58d8-4328-a73d-b80acd4342ce None admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:59.660654 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe1418a6-58d8-4328-a73d-b80acd4342ce None admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:59.660953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe1418a6-58d8-4328-a73d-b80acd4342ce None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:59.662362 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe1418a6-58d8-4328-a73d-b80acd4342ce None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:59.667727 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e76d458-b3a6-46b2-8a09-e59f8f9c7dff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:59.668343 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e76d458-b3a6-46b2-8a09-e59f8f9c7dff None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:59.668624 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e76d458-b3a6-46b2-8a09-e59f8f9c7dff None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:59.668895 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e76d458-b3a6-46b2-8a09-e59f8f9c7dff None admin] PATH_INFO: `/v3/projects/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:59.673325 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 259/519] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:11:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:59.674102 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe1418a6-58d8-4328-a73d-b80acd4342ce None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:59.676435 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e76d458-b3a6-46b2-8a09-e59f8f9c7dff None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:59.677773 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e76d458-b3a6-46b2-8a09-e59f8f9c7dff None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:59.678269 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:59.689750 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-3e76d458-b3a6-46b2-8a09-e59f8f9c7dff None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:11:59.689750 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:59.689750 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:59.689750 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:59.689750 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:59.689750 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:59.689750 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:59.689750 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:59.689750 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:59.689750 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:59.689750 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:59.689750 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:59.689750 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 27 01:11:59.689750 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 27 01:11:59.689750 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 27 01:11:59.690659 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 27 01:11:59.690659 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:59.690659 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:59.690659 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:59.690659 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:59.690659 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:59.690659 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:59.690659 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:59.690659 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:59.690659 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:59.690659 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:59.690659 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:59.690659 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:59.690659 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:59.690659 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:59.690659 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:59.690659 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:59.692380 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 27 01:11:59.692380 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 27 01:11:59.692380 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 27 01:11:59.692380 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 27 01:11:59.692380 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 27 01:11:59.692380 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 27 01:11:59.692380 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:11:59.692380 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:11:59.692380 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 261/520] 10.222.0.42 () {56 vars in 1111 bytes} [Thu Oct 27 01:11:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:59.692862 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e76d458-b3a6-46b2-8a09-e59f8f9c7dff None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:59.733226 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:59.776808 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe1418a6-58d8-4328-a73d-b80acd4342ce None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:59.789574 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e76d458-b3a6-46b2-8a09-e59f8f9c7dff None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:59.809900 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010bda3c-789f-43b8-acfc-5d9c4a89d7c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:59.810525 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010bda3c-789f-43b8-acfc-5d9c4a89d7c1 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:59.811119 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010bda3c-789f-43b8-acfc-5d9c4a89d7c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:59.811119 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010bda3c-789f-43b8-acfc-5d9c4a89d7c1 None admin] PATH_INFO: `/v3/services/network` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:59.811609 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010bda3c-789f-43b8-acfc-5d9c4a89d7c1 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:59.812960 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010bda3c-789f-43b8-acfc-5d9c4a89d7c1 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:59.824419 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a57ed4-d042-40f4-9568-f89bdf1442e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:59.825015 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a57ed4-d042-40f4-9568-f89bdf1442e2 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:59.825321 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a57ed4-d042-40f4-9568-f89bdf1442e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:59.825495 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-010bda3c-789f-43b8-acfc-5d9c4a89d7c1 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 27 01:11:59.825495 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:59.825495 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 27 01:11:59.825495 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 27 01:11:59.825495 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 27 01:11:59.825495 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 27 01:11:59.825495 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 27 01:11:59.825495 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:11:59.825495 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 27 01:11:59.825495 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:59.825495 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 27 01:11:59.825495 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:59.825495 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:11:59.825495 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:11:59.825495 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:11:59.825495 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:11:59.825495 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:11:59.826620 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:11:59.826620 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:11:59.826620 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:11:59.826620 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:11:59.826620 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:11:59.826620 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:11:59.826620 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 27 01:11:59.826620 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 27 01:11:59.826620 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 27 01:11:59.826620 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 27 01:11:59.826620 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:11:59.826620 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:11:59.826620 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:11:59.826620 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:11:59.826620 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:11:59.826620 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:11:59.827451 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:11:59.827451 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:11:59.827451 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:11:59.827451 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:11:59.827451 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:11:59.827451 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:11:59.827451 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:11:59.827451 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:11:59.827451 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:11:59.827451 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:11:59.827451 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 27 01:11:59.827451 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:11:59.827451 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 27 01:11:59.827451 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:11:59.828284 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a57ed4-d042-40f4-9568-f89bdf1442e2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:59.828284 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a57ed4-d042-40f4-9568-f89bdf1442e2 None admin] RBAC: Adding query filter params (role.id=1b8f8dd9e8584053b048032db63254a4, scope.project.id=23aad848d47c4d27ad4c138ff869e40f, user.id=77f2cf9e0c3541f9bccfb7e213d88609) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:59.828284 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a57ed4-d042-40f4-9568-f89bdf1442e2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:59.828591 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 260/521] 10.222.0.42 () {56 vars in 1111 bytes} [Thu Oct 27 01:11:59 2022] GET /identity/v3/services/network => generated 88 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:11:59.828591 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010bda3c-789f-43b8-acfc-5d9c4a89d7c1 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:59.828725 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a57ed4-d042-40f4-9568-f89bdf1442e2 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:59.835233 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 262/522] 10.222.0.42 () {56 vars in 1374 bytes} [Thu Oct 27 01:11:59 2022] GET /identity/v3/role_assignments?user.id=77f2cf9e0c3541f9bccfb7e213d88609&scope.project.id=23aad848d47c4d27ad4c138ff869e40f&role.id=1b8f8dd9e8584053b048032db63254a4 => generated 608 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:59.836005 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a57ed4-d042-40f4-9568-f89bdf1442e2 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:11:59.902954 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:59.941127 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010bda3c-789f-43b8-acfc-5d9c4a89d7c1 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:59.957580 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a57ed4-d042-40f4-9568-f89bdf1442e2 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:11:59.966694 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:11:59.980005 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38718e03-714e-425f-be1a-77a8c6d1af05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:11:59.980628 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38718e03-714e-425f-be1a-77a8c6d1af05 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:11:59.980907 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38718e03-714e-425f-be1a-77a8c6d1af05 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:11:59.981211 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38718e03-714e-425f-be1a-77a8c6d1af05 None admin] PATH_INFO: `/v3/services` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:11:59.982393 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38718e03-714e-425f-be1a-77a8c6d1af05 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:11:59.982799 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38718e03-714e-425f-be1a-77a8c6d1af05 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:11:59.984183 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38718e03-714e-425f-be1a-77a8c6d1af05 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:11:59.990066 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 261/523] 10.222.0.42 () {56 vars in 1124 bytes} [Thu Oct 27 01:11:59 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:11:59.990916 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38718e03-714e-425f-be1a-77a8c6d1af05 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:00.000879 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92156d70-7eb1-4625-bf9b-44248b0cc5bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:00.001752 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92156d70-7eb1-4625-bf9b-44248b0cc5bf None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:00.001863 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92156d70-7eb1-4625-bf9b-44248b0cc5bf None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:00.002059 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92156d70-7eb1-4625-bf9b-44248b0cc5bf None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:00.002704 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92156d70-7eb1-4625-bf9b-44248b0cc5bf None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:00.004781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92156d70-7eb1-4625-bf9b-44248b0cc5bf None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:00.027497 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 263/524] 10.222.0.42 () {60 vars in 1132 bytes} [Thu Oct 27 01:11:59 2022] POST /identity/v3/users => generated 254 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:12:00.028452 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92156d70-7eb1-4625-bf9b-44248b0cc5bf None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:00.132444 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38718e03-714e-425f-be1a-77a8c6d1af05 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:00.168557 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92156d70-7eb1-4625-bf9b-44248b0cc5bf None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:00.173386 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:00.181641 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29310fc-b7b6-46e3-b5cd-21b8d559638f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:00.182265 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29310fc-b7b6-46e3-b5cd-21b8d559638f None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:00.182535 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29310fc-b7b6-46e3-b5cd-21b8d559638f None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:00.183206 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29310fc-b7b6-46e3-b5cd-21b8d559638f None admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:00.183945 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d29310fc-b7b6-46e3-b5cd-21b8d559638f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:00.184249 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d29310fc-b7b6-46e3-b5cd-21b8d559638f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:00.186138 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d29310fc-b7b6-46e3-b5cd-21b8d559638f None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:00.206405 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 262/525] 10.222.0.42 () {56 vars in 1148 bytes} [Thu Oct 27 01:11:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:00.206405 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29310fc-b7b6-46e3-b5cd-21b8d559638f None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:00.224673 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9782422-e730-41ad-96d4-54ba301b58d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:00.225604 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9782422-e730-41ad-96d4-54ba301b58d7 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:00.226832 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9782422-e730-41ad-96d4-54ba301b58d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:00.227257 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9782422-e730-41ad-96d4-54ba301b58d7 None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:00.228033 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9782422-e730-41ad-96d4-54ba301b58d7 None admin] RBAC: Adding query filter params (name=network) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:00.229313 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9782422-e730-41ad-96d4-54ba301b58d7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:00.230706 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9782422-e730-41ad-96d4-54ba301b58d7 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:00.240731 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 264/526] 10.222.0.42 () {56 vars in 1112 bytes} [Thu Oct 27 01:12:00 2022] GET /identity/v3/services?name=network => generated 124 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:00.242140 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9782422-e730-41ad-96d4-54ba301b58d7 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:00.277611 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:00.340954 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29310fc-b7b6-46e3-b5cd-21b8d559638f None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:00.368315 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9782422-e730-41ad-96d4-54ba301b58d7 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:00.388525 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e1c2a9-cd8c-4dc0-adb8-f195dd7578c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:00.389413 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e1c2a9-cd8c-4dc0-adb8-f195dd7578c8 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:00.389700 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e1c2a9-cd8c-4dc0-adb8-f195dd7578c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:00.389970 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e1c2a9-cd8c-4dc0-adb8-f195dd7578c8 None admin] PATH_INFO: `/v3/services` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:00.391075 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e1c2a9-cd8c-4dc0-adb8-f195dd7578c8 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:00.391421 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e1c2a9-cd8c-4dc0-adb8-f195dd7578c8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:00.392793 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e1c2a9-cd8c-4dc0-adb8-f195dd7578c8 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:00.401177 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 263/527] 10.222.0.42 () {56 vars in 1124 bytes} [Thu Oct 27 01:12:00 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:00.402187 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e1c2a9-cd8c-4dc0-adb8-f195dd7578c8 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:00.410953 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4593a081-ac3d-4e81-afea-2e6f20fdf5b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:00.411629 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4593a081-ac3d-4e81-afea-2e6f20fdf5b7 None admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:00.411963 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4593a081-ac3d-4e81-afea-2e6f20fdf5b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:00.412241 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4593a081-ac3d-4e81-afea-2e6f20fdf5b7 None admin] PATH_INFO: `/v3/projects/23aad848d47c4d27ad4c138ff869e40f/users/51033d56f85f4282a49ded6338440677/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:00.414852 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4593a081-ac3d-4e81-afea-2e6f20fdf5b7 None admin] RBAC: Authorizing `identity:create_grant(project_id=23aad848d47c4d27ad4c138ff869e40f, user_id=51033d56f85f4282a49ded6338440677, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:00.416543 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4593a081-ac3d-4e81-afea-2e6f20fdf5b7 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:00.433270 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 265/528] 10.222.0.42 () {58 vars in 1439 bytes} [Thu Oct 27 01:12:00 2022] PUT /identity/v3/projects/23aad848d47c4d27ad4c138ff869e40f/users/51033d56f85f4282a49ded6338440677/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:12:00.486513 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:00.570005 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e1c2a9-cd8c-4dc0-adb8-f195dd7578c8 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:00.612693 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021b7d94-ef59-449d-8e0b-059f6469c7c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:00.614156 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021b7d94-ef59-449d-8e0b-059f6469c7c8 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:00.614435 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021b7d94-ef59-449d-8e0b-059f6469c7c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:00.614701 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021b7d94-ef59-449d-8e0b-059f6469c7c8 None admin] PATH_INFO: `/v3/services` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:00.615344 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021b7d94-ef59-449d-8e0b-059f6469c7c8 None admin] RBAC: Adding query filter params (type=network) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:00.615643 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021b7d94-ef59-449d-8e0b-059f6469c7c8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:00.617048 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021b7d94-ef59-449d-8e0b-059f6469c7c8 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:00.622599 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 264/529] 10.222.0.42 () {56 vars in 1112 bytes} [Thu Oct 27 01:12:00 2022] GET /identity/v3/services?type=network => generated 124 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:02.337079 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eab5e0e-a1c7-4c2f-8b19-719b329bf26a None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:02.338123 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eab5e0e-a1c7-4c2f-8b19-719b329bf26a None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:02.338607 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eab5e0e-a1c7-4c2f-8b19-719b329bf26a None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:02.340432 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 266/530] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:02.346020 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f182abc-7ebc-48ce-9278-d3985c1a4f08 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:02.346842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f182abc-7ebc-48ce-9278-d3985c1a4f08 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:02.347988 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f182abc-7ebc-48ce-9278-d3985c1a4f08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:02.373883 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-5f182abc-7ebc-48ce-9278-d3985c1a4f08 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:02.375404 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-5f182abc-7ebc-48ce-9278-d3985c1a4f08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:02.465489 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 265/531] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 27 01:12:02.741117 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1a4828-5960-4e30-9d46-3ee39e60dedd None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:02.741830 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1a4828-5960-4e30-9d46-3ee39e60dedd None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:02.742327 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1a4828-5960-4e30-9d46-3ee39e60dedd None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:02.744120 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 267/532] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:02.749264 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e1d5ad-cd68-41e0-a00b-a84b43d191f3 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:02.749552 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e1d5ad-cd68-41e0-a00b-a84b43d191f3 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:02.749827 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e1d5ad-cd68-41e0-a00b-a84b43d191f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:02.774306 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-e8e1d5ad-cd68-41e0-a00b-a84b43d191f3 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:02.775480 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-e8e1d5ad-cd68-41e0-a00b-a84b43d191f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:02.843481 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:02.848981 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1a4828-5960-4e30-9d46-3ee39e60dedd None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:02.876864 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 266/533] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:12:02.964080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:02.983671 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1a4828-5960-4e30-9d46-3ee39e60dedd None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:03.024752 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edee9a54-7a63-45ce-9d07-489f9034d088 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:03.025352 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edee9a54-7a63-45ce-9d07-489f9034d088 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:03.025631 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edee9a54-7a63-45ce-9d07-489f9034d088 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:03.025907 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edee9a54-7a63-45ce-9d07-489f9034d088 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:03.049095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edee9a54-7a63-45ce-9d07-489f9034d088 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:12:03.054054 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-edee9a54-7a63-45ce-9d07-489f9034d088 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:12:03.054054 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:03.054054 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:03.054054 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:03.054054 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:03.054054 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:03.054054 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:03.054054 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:03.054054 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:03.054054 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:03.054054 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:03.054054 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:03.054054 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:12:03.054054 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:12:03.054054 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:12:03.054054 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:12:03.054987 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:12:03.054987 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:12:03.054987 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:12:03.054987 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:12:03.054987 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:03.054987 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:03.054987 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:03.054987 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:03.054987 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:03.054987 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:03.054987 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:03.054987 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:03.054987 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:03.054987 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:03.054987 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:03.054987 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:03.055851 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:03.055851 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:03.055851 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:03.055851 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:03.055851 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:12:03.055851 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:12:03.055851 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:12:03.055851 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:12:03.055851 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:12:03.055851 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:12:03.055851 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:12:03.055851 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:03.055851 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 268/534] 10.222.0.42 () {56 vars in 1096 bytes} [Thu Oct 27 01:12:02 2022] GET /identity/v3/roles/admin => generated 83 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:03.063101 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e1d5ad-cd68-41e0-a00b-a84b43d191f3 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:03.184477 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1466b54-2f02-4a46-8ea6-49299f9c2125 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:03.184797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1466b54-2f02-4a46-8ea6-49299f9c2125 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:03.185022 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1466b54-2f02-4a46-8ea6-49299f9c2125 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:03.186513 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 269/535] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:03.199737 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1466b54-2f02-4a46-8ea6-49299f9c2125 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:03.209971 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e1d5ad-cd68-41e0-a00b-a84b43d191f3 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:03.247687 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c068c1-de58-41b7-84d6-7d89cf702e4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:03.248291 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c068c1-de58-41b7-84d6-7d89cf702e4a None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:03.248923 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c068c1-de58-41b7-84d6-7d89cf702e4a None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:03.249236 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c068c1-de58-41b7-84d6-7d89cf702e4a None admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:03.250244 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c068c1-de58-41b7-84d6-7d89cf702e4a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:03.250549 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c068c1-de58-41b7-84d6-7d89cf702e4a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:03.252083 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c068c1-de58-41b7-84d6-7d89cf702e4a None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:03.265934 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 267/536] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:12:03 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:03.265934 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ad7972-47c3-48b8-a992-4ea5368f2955 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:03.265934 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ad7972-47c3-48b8-a992-4ea5368f2955 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:03.266239 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ad7972-47c3-48b8-a992-4ea5368f2955 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:03.284306 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-79ad7972-47c3-48b8-a992-4ea5368f2955 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:03.285796 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-79ad7972-47c3-48b8-a992-4ea5368f2955 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:03.318782 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1466b54-2f02-4a46-8ea6-49299f9c2125 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:03.325237 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:03.355440 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d3036bc-ca7f-4d8f-9003-16c681a1a9ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:03.356072 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3036bc-ca7f-4d8f-9003-16c681a1a9ce None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:03.356350 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3036bc-ca7f-4d8f-9003-16c681a1a9ce None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:03.356624 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3036bc-ca7f-4d8f-9003-16c681a1a9ce None admin] PATH_INFO: `/v3/roles/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:03.362026 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 268/537] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:12:03.364445 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fe606d-02e3-4693-a1aa-dfe7b9f5eb04 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:03.364962 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fe606d-02e3-4693-a1aa-dfe7b9f5eb04 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:03.365437 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fe606d-02e3-4693-a1aa-dfe7b9f5eb04 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:03.367253 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 269/538] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:03.368216 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03fe606d-02e3-4693-a1aa-dfe7b9f5eb04 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:03.389459 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d3036bc-ca7f-4d8f-9003-16c681a1a9ce None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:12:03.396173 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-4d3036bc-ca7f-4d8f-9003-16c681a1a9ce None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 27 01:12:03.396173 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:03.396173 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:03.396173 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:03.396173 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:03.396173 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:03.396173 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:03.396173 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:03.396173 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:03.396173 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:03.396173 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:03.396173 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:03.396173 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:12:03.396173 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:12:03.396173 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:12:03.396173 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:12:03.397194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:12:03.397194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:12:03.397194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:12:03.397194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:12:03.397194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:03.397194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:03.397194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:03.397194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:03.397194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:03.397194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:03.397194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:03.397194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:03.397194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:03.397194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:03.397194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:03.397194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:03.398047 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:03.398047 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:03.398047 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:03.398047 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:03.398047 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:12:03.398047 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:12:03.398047 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:12:03.398047 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:12:03.398047 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:12:03.398047 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:12:03.398047 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 27 01:12:03.398047 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:03.401200 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 270/539] 10.222.0.42 () {56 vars in 1102 bytes} [Thu Oct 27 01:12:03 2022] GET /identity/v3/roles/service => generated 85 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:03.403950 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43dc2ea-9184-438b-a611-367ae4b86212 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:03.404237 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43dc2ea-9184-438b-a611-367ae4b86212 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:03.404512 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43dc2ea-9184-438b-a611-367ae4b86212 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:03.417219 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:03.422924 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-a43dc2ea-9184-438b-a611-367ae4b86212 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:03.424240 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-a43dc2ea-9184-438b-a611-367ae4b86212 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:03.507013 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03fe606d-02e3-4693-a1aa-dfe7b9f5eb04 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:03.519433 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 271/540] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:12:03.521358 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43dc2ea-9184-438b-a611-367ae4b86212 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:03.544775 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de7a5221-9bed-4340-b3e2-41725f237453 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:03.545907 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7a5221-9bed-4340-b3e2-41725f237453 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:03.546370 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7a5221-9bed-4340-b3e2-41725f237453 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:03.546903 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7a5221-9bed-4340-b3e2-41725f237453 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:03.559799 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de7a5221-9bed-4340-b3e2-41725f237453 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:03.561863 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de7a5221-9bed-4340-b3e2-41725f237453 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:03.582996 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-de7a5221-9bed-4340-b3e2-41725f237453 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:12:03.582996 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:03.582996 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:03.582996 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:03.582996 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:03.582996 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:03.582996 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:03.582996 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:03.582996 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:03.582996 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:03.582996 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:03.582996 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:03.582996 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:12:03.582996 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:12:03.582996 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:12:03.582996 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:12:03.584065 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:03.584065 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:03.584065 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:03.584065 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:03.584065 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:03.584065 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:03.584065 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:03.584065 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:03.584065 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:03.584065 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:03.584065 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:03.584065 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:03.584065 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:03.584065 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:03.584065 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:03.584065 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:03.584065 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:12:03.585007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:12:03.585007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:12:03.585007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:03.587568 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 270/541] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:12:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:03.596458 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de7a5221-9bed-4340-b3e2-41725f237453 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:03.644708 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:03.655805 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43dc2ea-9184-438b-a611-367ae4b86212 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:03.697691 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da05db89-a73b-4720-8a08-5d3333c5490a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:03.698298 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da05db89-a73b-4720-8a08-5d3333c5490a None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:03.698575 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da05db89-a73b-4720-8a08-5d3333c5490a None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:03.698844 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da05db89-a73b-4720-8a08-5d3333c5490a None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:03.699511 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da05db89-a73b-4720-8a08-5d3333c5490a None admin] RBAC: Adding query filter params (name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:03.699922 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da05db89-a73b-4720-8a08-5d3333c5490a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:03.701301 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da05db89-a73b-4720-8a08-5d3333c5490a None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:03.703782 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:03.714899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 272/542] 10.222.0.42 () {56 vars in 1103 bytes} [Thu Oct 27 01:12:03 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:03.714899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da05db89-a73b-4720-8a08-5d3333c5490a None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:03.751465 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de7a5221-9bed-4340-b3e2-41725f237453 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:03.790727 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aecf8bf3-3ab2-476f-a521-6cc48010c7aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:03.791761 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aecf8bf3-3ab2-476f-a521-6cc48010c7aa None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:03.792111 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aecf8bf3-3ab2-476f-a521-6cc48010c7aa None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:03.792414 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aecf8bf3-3ab2-476f-a521-6cc48010c7aa None admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:03.793090 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aecf8bf3-3ab2-476f-a521-6cc48010c7aa None admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:03.793393 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aecf8bf3-3ab2-476f-a521-6cc48010c7aa None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:03.795499 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aecf8bf3-3ab2-476f-a521-6cc48010c7aa None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:03.808791 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 271/543] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:12:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:03.808791 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b503107-4463-47be-a8fd-5bac1d6f72cd None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:03.808791 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b503107-4463-47be-a8fd-5bac1d6f72cd None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:03.808791 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b503107-4463-47be-a8fd-5bac1d6f72cd None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:03.808791 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 272/544] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:03.808791 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b503107-4463-47be-a8fd-5bac1d6f72cd None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:03.829753 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da05db89-a73b-4720-8a08-5d3333c5490a None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:03.872496 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5058dc-0c4f-48c5-9f00-03e3d994fd77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:03.873116 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5058dc-0c4f-48c5-9f00-03e3d994fd77 None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:03.873394 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5058dc-0c4f-48c5-9f00-03e3d994fd77 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:03.873835 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5058dc-0c4f-48c5-9f00-03e3d994fd77 None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:03.874804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c5058dc-0c4f-48c5-9f00-03e3d994fd77 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:03.876167 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c5058dc-0c4f-48c5-9f00-03e3d994fd77 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:03.893105 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 273/545] 10.222.0.42 () {60 vars in 1141 bytes} [Thu Oct 27 01:12:03 2022] POST /identity/v3/services => generated 251 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:12:03.893376 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5058dc-0c4f-48c5-9f00-03e3d994fd77 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:03.910771 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:03.939826 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b503107-4463-47be-a8fd-5bac1d6f72cd None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:03.963560 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:03.980428 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfd0d72-b77f-42d7-805a-d9dc4f5ccbbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:03.981242 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfd0d72-b77f-42d7-805a-d9dc4f5ccbbc None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:03.981521 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfd0d72-b77f-42d7-805a-d9dc4f5ccbbc None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:03.981789 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfd0d72-b77f-42d7-805a-d9dc4f5ccbbc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:03.993217 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dfd0d72-b77f-42d7-805a-d9dc4f5ccbbc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:03.994670 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dfd0d72-b77f-42d7-805a-d9dc4f5ccbbc None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:04.010533 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-7dfd0d72-b77f-42d7-805a-d9dc4f5ccbbc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:12:04.010533 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:04.010533 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:04.010533 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:04.010533 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:04.010533 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:04.010533 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:04.010533 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:04.010533 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:04.010533 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:04.010533 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:04.010533 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:04.010533 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:12:04.010533 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:12:04.010533 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:12:04.010533 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:12:04.011789 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:04.011789 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:04.011789 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:04.011789 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:04.011789 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:04.011789 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:04.011789 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:04.011789 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:04.011789 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:04.011789 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:04.011789 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:04.011789 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:04.011789 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:04.011789 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:04.011789 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:04.011789 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:04.011789 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:12:04.013367 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:12:04.013367 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:12:04.013367 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:04.016534 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 273/546] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:12:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:04.017954 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad473f5-6364-40b4-af3c-85e60cfaf9e2 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:04.018180 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5058dc-0c4f-48c5-9f00-03e3d994fd77 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:04.018287 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad473f5-6364-40b4-af3c-85e60cfaf9e2 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:04.018491 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad473f5-6364-40b4-af3c-85e60cfaf9e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:04.037550 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-aad473f5-6364-40b4-af3c-85e60cfaf9e2 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:04.038758 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-aad473f5-6364-40b4-af3c-85e60cfaf9e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:04.059027 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6681738a-2a85-4e4f-afbb-d0cc93a34421 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:04.059684 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6681738a-2a85-4e4f-afbb-d0cc93a34421 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:04.060599 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6681738a-2a85-4e4f-afbb-d0cc93a34421 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:04.060960 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6681738a-2a85-4e4f-afbb-d0cc93a34421 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:04.071045 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6681738a-2a85-4e4f-afbb-d0cc93a34421 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:04.072495 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6681738a-2a85-4e4f-afbb-d0cc93a34421 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:04.084471 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-6681738a-2a85-4e4f-afbb-d0cc93a34421 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 27 01:12:04.084471 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:04.084471 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:04.084471 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:04.084471 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:04.084471 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:04.084471 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:04.084471 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:04.084471 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:04.084471 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:04.084471 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:04.084471 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:04.084471 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 27 01:12:04.084471 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 27 01:12:04.084471 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 27 01:12:04.084471 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 27 01:12:04.085696 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:04.085696 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:04.085696 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 27 01:12:04.085696 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:12:04.085696 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 27 01:12:04.085696 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:12:04.085696 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:04.085696 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:04.085696 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:04.085696 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:04.085696 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:04.085696 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:04.085696 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:04.085696 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:04.085696 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:04.085696 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:04.085696 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:04.085696 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:04.086891 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:04.086891 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:04.086891 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 27 01:12:04.086891 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 27 01:12:04.086891 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 27 01:12:04.086891 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 27 01:12:04.086891 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 27 01:12:04.086891 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 27 01:12:04.086891 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 27 01:12:04.086891 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:04.086891 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 274/547] 10.222.0.42 () {56 vars in 1096 bytes} [Thu Oct 27 01:12:03 2022] GET /identity/v3/users/swift => generated 83 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:04.086891 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6681738a-2a85-4e4f-afbb-d0cc93a34421 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:04.118149 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 274/548] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:04 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 27 01:12:04.118967 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad473f5-6364-40b4-af3c-85e60cfaf9e2 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:04.197403 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6681738a-2a85-4e4f-afbb-d0cc93a34421 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:04.204731 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:04.217573 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:04.241553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55bc5b0e-3b74-4f51-9b80-017004744772 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:04.242693 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bc5b0e-3b74-4f51-9b80-017004744772 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:04.243119 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bc5b0e-3b74-4f51-9b80-017004744772 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:04.243612 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bc5b0e-3b74-4f51-9b80-017004744772 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:04.250091 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad473f5-6364-40b4-af3c-85e60cfaf9e2 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:04.256019 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55bc5b0e-3b74-4f51-9b80-017004744772 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:12:04.261867 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-55bc5b0e-3b74-4f51-9b80-017004744772 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:12:04.261867 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:04.261867 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:04.261867 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:04.261867 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:04.261867 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:04.261867 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:04.261867 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:04.261867 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:04.261867 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:04.261867 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:04.261867 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:04.261867 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:12:04.261867 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:12:04.261867 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:12:04.261867 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:12:04.263018 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:12:04.263018 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:12:04.263018 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:12:04.263018 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:12:04.263018 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:04.263018 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:04.263018 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:04.263018 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:04.263018 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:04.263018 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:04.263018 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:04.263018 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:04.263018 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:04.263018 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:04.263018 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:04.263018 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:04.264054 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:04.264054 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:04.264054 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:04.264054 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:04.264054 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:12:04.264054 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:12:04.264054 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:12:04.264054 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:12:04.264054 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:12:04.264054 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:12:04.264054 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:12:04.264054 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:04.267169 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 275/549] 10.222.0.42 () {56 vars in 1096 bytes} [Thu Oct 27 01:12:04 2022] GET /identity/v3/roles/admin => generated 83 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:04.268178 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55bc5b0e-3b74-4f51-9b80-017004744772 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:04.292897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3de0703-17e5-4b9f-839e-819528e882c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:04.293671 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3de0703-17e5-4b9f-839e-819528e882c0 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:04.294021 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3de0703-17e5-4b9f-839e-819528e882c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:04.294359 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3de0703-17e5-4b9f-839e-819528e882c0 None admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:04.295121 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3de0703-17e5-4b9f-839e-819528e882c0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:04.295674 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3de0703-17e5-4b9f-839e-819528e882c0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:04.297020 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3de0703-17e5-4b9f-839e-819528e882c0 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:04.310852 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 275/550] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:12:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:04.311237 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3de0703-17e5-4b9f-839e-819528e882c0 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:04.392132 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55bc5b0e-3b74-4f51-9b80-017004744772 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:04.416262 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3de0703-17e5-4b9f-839e-819528e882c0 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:04.430575 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cce06fd-50b0-46d9-b344-6900ff2818e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:04.431169 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cce06fd-50b0-46d9-b344-6900ff2818e7 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:04.431725 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cce06fd-50b0-46d9-b344-6900ff2818e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:04.431983 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cce06fd-50b0-46d9-b344-6900ff2818e7 None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:04.432714 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cce06fd-50b0-46d9-b344-6900ff2818e7 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:04.433016 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cce06fd-50b0-46d9-b344-6900ff2818e7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:04.434477 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cce06fd-50b0-46d9-b344-6900ff2818e7 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:04.441258 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:04.458053 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 276/551] 10.222.0.42 () {56 vars in 1135 bytes} [Thu Oct 27 01:12:04 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:04.458053 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cce06fd-50b0-46d9-b344-6900ff2818e7 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:04.459220 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d785edc-b1c7-4948-ad18-1e45fa2d6ea7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:04.459840 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d785edc-b1c7-4948-ad18-1e45fa2d6ea7 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:04.460117 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d785edc-b1c7-4948-ad18-1e45fa2d6ea7 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:04.460389 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d785edc-b1c7-4948-ad18-1e45fa2d6ea7 None admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:04.461074 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d785edc-b1c7-4948-ad18-1e45fa2d6ea7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:04.461387 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d785edc-b1c7-4948-ad18-1e45fa2d6ea7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:04.462757 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d785edc-b1c7-4948-ad18-1e45fa2d6ea7 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:04.472886 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 276/552] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:12:04 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:04.473636 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d785edc-b1c7-4948-ad18-1e45fa2d6ea7 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:04.499400 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:04.570183 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cce06fd-50b0-46d9-b344-6900ff2818e7 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:04.581838 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d785edc-b1c7-4948-ad18-1e45fa2d6ea7 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:04.610324 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f59ad2-cca1-45de-be79-e87cd8cf7fb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:04.611626 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f59ad2-cca1-45de-be79-e87cd8cf7fb9 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:04.612496 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f59ad2-cca1-45de-be79-e87cd8cf7fb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:04.613727 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f59ad2-cca1-45de-be79-e87cd8cf7fb9 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:04.617494 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08aa1a9-350e-41d4-89be-a81544e3633d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:04.618098 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08aa1a9-350e-41d4-89be-a81544e3633d None admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:04.618372 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08aa1a9-350e-41d4-89be-a81544e3633d None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:04.618639 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08aa1a9-350e-41d4-89be-a81544e3633d None admin] PATH_INFO: `/v3/services` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:04.619282 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d08aa1a9-350e-41d4-89be-a81544e3633d None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:04.620646 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d08aa1a9-350e-41d4-89be-a81544e3633d None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:04.620988 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f59ad2-cca1-45de-be79-e87cd8cf7fb9 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:04.622452 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f59ad2-cca1-45de-be79-e87cd8cf7fb9 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:04.635299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-89f59ad2-cca1-45de-be79-e87cd8cf7fb9 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 27 01:12:04.635299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:04.635299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:04.635299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:04.635299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:04.635299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:04.635299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:04.635299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:04.635299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:04.635299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:04.635299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:04.635299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:04.635299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 27 01:12:04.635299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 27 01:12:04.635299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 27 01:12:04.635299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 27 01:12:04.636544 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:04.636544 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:04.636544 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 27 01:12:04.636544 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:12:04.636544 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 27 01:12:04.636544 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:12:04.636544 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:04.636544 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:04.636544 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:04.636544 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:04.636544 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:04.636544 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:04.636544 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:04.636544 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:04.636544 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:04.636544 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:04.636544 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:04.636544 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:04.637904 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:04.637904 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:04.637904 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 27 01:12:04.637904 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 27 01:12:04.637904 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 27 01:12:04.637904 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 27 01:12:04.637904 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 27 01:12:04.637904 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 27 01:12:04.637904 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 27 01:12:04.637904 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:04.645101 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 277/553] 10.222.0.42 () {60 vars in 1141 bytes} [Thu Oct 27 01:12:04 2022] POST /identity/v3/services => generated 240 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:12:04.645101 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08aa1a9-350e-41d4-89be-a81544e3633d None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:04.645362 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 277/554] 10.222.0.42 () {56 vars in 1117 bytes} [Thu Oct 27 01:12:04 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:04.645362 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f59ad2-cca1-45de-be79-e87cd8cf7fb9 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:04.729523 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:04.773103 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:04.780018 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f59ad2-cca1-45de-be79-e87cd8cf7fb9 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:04.801672 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08aa1a9-350e-41d4-89be-a81544e3633d None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:04.835022 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff54fb61-cf5d-48ec-8540-eb847c81201b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:04.835911 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff54fb61-cf5d-48ec-8540-eb847c81201b None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:04.836330 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff54fb61-cf5d-48ec-8540-eb847c81201b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:04.836737 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff54fb61-cf5d-48ec-8540-eb847c81201b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:04.854678 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff54fb61-cf5d-48ec-8540-eb847c81201b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:04.856180 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff54fb61-cf5d-48ec-8540-eb847c81201b None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:04.857943 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf30228-870f-4fcd-9901-5ee67ae980de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:04.858592 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf30228-870f-4fcd-9901-5ee67ae980de None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:04.858865 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf30228-870f-4fcd-9901-5ee67ae980de None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:04.859260 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf30228-870f-4fcd-9901-5ee67ae980de None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:04.871476 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf30228-870f-4fcd-9901-5ee67ae980de None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:04.872916 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf30228-870f-4fcd-9901-5ee67ae980de None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:04.875765 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-ff54fb61-cf5d-48ec-8540-eb847c81201b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:12:04.875765 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:04.875765 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:04.875765 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:04.875765 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:04.875765 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:04.875765 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:04.875765 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:04.875765 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:04.875765 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:04.875765 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:04.875765 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:04.875765 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:12:04.875765 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:12:04.875765 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:12:04.875765 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:12:04.876736 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:04.876736 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:04.876736 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:04.876736 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:04.876736 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:04.876736 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:04.876736 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:04.876736 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:04.876736 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:04.876736 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:04.876736 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:04.876736 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:04.876736 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:04.876736 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:04.876736 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:04.876736 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:04.876736 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:12:04.877883 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:12:04.877883 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:12:04.877883 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:04.879371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 278/555] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:12:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:04.880281 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff54fb61-cf5d-48ec-8540-eb847c81201b None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:04.892957 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-fdf30228-870f-4fcd-9901-5ee67ae980de None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:12:04.892957 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:04.892957 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:04.892957 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:04.892957 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:04.892957 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:04.892957 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:04.892957 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:04.892957 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:04.892957 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:04.892957 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:04.892957 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:04.892957 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:12:04.892957 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:12:04.892957 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:12:04.892957 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:12:04.894626 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:04.894626 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:04.894626 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:04.894626 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:04.894626 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:04.894626 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:04.894626 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:04.894626 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:04.894626 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:04.894626 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:04.894626 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:04.894626 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:04.894626 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:04.894626 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:04.894626 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:04.894626 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:04.894626 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:12:04.895673 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:12:04.895673 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:12:04.895673 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:04.898153 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 278/556] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:12:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:04.898153 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf30228-870f-4fcd-9901-5ee67ae980de None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:04.990924 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff54fb61-cf5d-48ec-8540-eb847c81201b None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:05.009729 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:05.017101 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf30228-870f-4fcd-9901-5ee67ae980de None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:05.028796 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a675b566-786b-4338-858c-47e953f4fb7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:05.029444 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a675b566-786b-4338-858c-47e953f4fb7b None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:05.029773 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a675b566-786b-4338-858c-47e953f4fb7b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:05.030049 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a675b566-786b-4338-858c-47e953f4fb7b None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:05.030787 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a675b566-786b-4338-858c-47e953f4fb7b None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:05.031425 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a675b566-786b-4338-858c-47e953f4fb7b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:05.032588 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a675b566-786b-4338-858c-47e953f4fb7b None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:05.053204 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 279/557] 10.222.0.42 () {56 vars in 1149 bytes} [Thu Oct 27 01:12:04 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:05.054209 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a675b566-786b-4338-858c-47e953f4fb7b None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:05.066701 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371783d6-4483-45d3-ae37-e403f9f8833b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:05.067700 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371783d6-4483-45d3-ae37-e403f9f8833b None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:05.068166 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:05.069365 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371783d6-4483-45d3-ae37-e403f9f8833b None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:05.070225 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371783d6-4483-45d3-ae37-e403f9f8833b None admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:05.071173 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371783d6-4483-45d3-ae37-e403f9f8833b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:05.071656 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371783d6-4483-45d3-ae37-e403f9f8833b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:05.074397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371783d6-4483-45d3-ae37-e403f9f8833b None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:05.090784 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 279/558] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:12:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:05.091831 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371783d6-4483-45d3-ae37-e403f9f8833b None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:05.177939 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a675b566-786b-4338-858c-47e953f4fb7b None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:05.209304 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371783d6-4483-45d3-ae37-e403f9f8833b None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:05.218850 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfeb58d-6a8e-4998-bd30-d3a9db9a8bfb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:05.219828 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfeb58d-6a8e-4998-bd30-d3a9db9a8bfb None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:05.220288 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfeb58d-6a8e-4998-bd30-d3a9db9a8bfb None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:05.220843 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfeb58d-6a8e-4998-bd30-d3a9db9a8bfb None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:05.221707 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccfeb58d-6a8e-4998-bd30-d3a9db9a8bfb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:05.222707 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccfeb58d-6a8e-4998-bd30-d3a9db9a8bfb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:05.224280 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccfeb58d-6a8e-4998-bd30-d3a9db9a8bfb None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:05.237247 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 280/559] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:12:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:05.238929 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfeb58d-6a8e-4998-bd30-d3a9db9a8bfb None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:05.259478 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d40100d-0c28-4183-af5d-d9006e630e67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:05.260292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d40100d-0c28-4183-af5d-d9006e630e67 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:05.260508 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d40100d-0c28-4183-af5d-d9006e630e67 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:05.260826 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d40100d-0c28-4183-af5d-d9006e630e67 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:05.274299 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d40100d-0c28-4183-af5d-d9006e630e67 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:05.276626 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d40100d-0c28-4183-af5d-d9006e630e67 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:05.297175 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-2d40100d-0c28-4183-af5d-d9006e630e67 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:12:05.297175 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:05.297175 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:05.297175 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:05.297175 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:05.297175 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:05.297175 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:05.297175 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:05.297175 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:05.297175 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:05.297175 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:05.297175 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:05.297175 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:12:05.297175 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:12:05.297175 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:12:05.297175 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:12:05.298208 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:05.298208 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:05.298208 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:05.298208 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:05.298208 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:05.298208 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:05.298208 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:05.298208 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:05.298208 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:05.298208 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:05.298208 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:05.298208 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:05.298208 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:05.298208 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:05.298208 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:05.298208 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:05.298208 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:12:05.299105 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:12:05.299105 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:12:05.299105 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:05.299105 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 280/560] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:12:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:05.301053 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d40100d-0c28-4183-af5d-d9006e630e67 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:05.321439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:05.321636 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:05.355935 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfeb58d-6a8e-4998-bd30-d3a9db9a8bfb None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:05.396600 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da3ff903-1eb2-4ef2-a393-59c314c131b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:05.398228 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3ff903-1eb2-4ef2-a393-59c314c131b4 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:05.398714 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3ff903-1eb2-4ef2-a393-59c314c131b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:05.399744 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3ff903-1eb2-4ef2-a393-59c314c131b4 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:05.413350 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da3ff903-1eb2-4ef2-a393-59c314c131b4 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:05.415190 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da3ff903-1eb2-4ef2-a393-59c314c131b4 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:05.428198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-da3ff903-1eb2-4ef2-a393-59c314c131b4 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 27 01:12:05.428198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:05.428198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:05.428198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:05.428198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:05.428198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:05.428198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:05.428198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:05.428198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:05.428198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:05.428198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:05.428198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:05.428198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 27 01:12:05.428198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 27 01:12:05.428198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 27 01:12:05.428198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 27 01:12:05.429250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:05.429250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:05.429250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 27 01:12:05.429250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:12:05.429250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 27 01:12:05.429250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:12:05.429250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:05.429250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:05.429250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:05.429250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:05.429250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:05.429250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:05.429250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:05.429250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:05.429250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:05.429250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:05.429250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:05.429250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:05.430204 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:05.430204 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:05.430204 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 27 01:12:05.430204 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 27 01:12:05.430204 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 27 01:12:05.430204 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 27 01:12:05.430204 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 27 01:12:05.430204 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 27 01:12:05.430204 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 27 01:12:05.430204 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:05.430744 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 281/561] 10.222.0.42 () {56 vars in 1093 bytes} [Thu Oct 27 01:12:05 2022] GET /identity/v3/users/nova => generated 82 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:05.433950 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da3ff903-1eb2-4ef2-a393-59c314c131b4 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:05.463476 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d40100d-0c28-4183-af5d-d9006e630e67 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:05.505922 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e2ffbfa-70f8-46c5-a541-13cf7d3ff596 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:05.506927 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2ffbfa-70f8-46c5-a541-13cf7d3ff596 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:05.507423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2ffbfa-70f8-46c5-a541-13cf7d3ff596 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:05.507888 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2ffbfa-70f8-46c5-a541-13cf7d3ff596 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:05.516283 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e2ffbfa-70f8-46c5-a541-13cf7d3ff596 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:05.518070 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e2ffbfa-70f8-46c5-a541-13cf7d3ff596 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:05.531717 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-4e2ffbfa-70f8-46c5-a541-13cf7d3ff596 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:12:05.531717 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:05.531717 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:05.531717 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:05.531717 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:05.531717 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:05.531717 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:05.531717 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:05.531717 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:05.531717 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:05.531717 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:05.531717 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:05.531717 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 27 01:12:05.531717 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 27 01:12:05.531717 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 27 01:12:05.532733 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 27 01:12:05.532733 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:05.532733 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:05.532733 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:05.532733 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:05.532733 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:05.532733 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:05.532733 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:05.532733 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:05.532733 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:05.532733 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:05.532733 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:05.532733 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:05.532733 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:05.532733 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:05.532733 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:05.532733 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:05.533684 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 27 01:12:05.533684 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 27 01:12:05.533684 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 27 01:12:05.533684 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 27 01:12:05.533684 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 27 01:12:05.533684 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 27 01:12:05.533684 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:12:05.533684 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:05.538090 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 281/562] 10.222.0.42 () {56 vars in 1111 bytes} [Thu Oct 27 01:12:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:05.539127 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e2ffbfa-70f8-46c5-a541-13cf7d3ff596 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:05.585263 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da3ff903-1eb2-4ef2-a393-59c314c131b4 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:05.613984 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:05.617888 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:05.631729 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-457bfd6c-2faa-45a9-a5eb-3fc2aac37bcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:05.632373 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457bfd6c-2faa-45a9-a5eb-3fc2aac37bcd None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:05.632656 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457bfd6c-2faa-45a9-a5eb-3fc2aac37bcd None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:05.632928 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457bfd6c-2faa-45a9-a5eb-3fc2aac37bcd None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:05.633617 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-457bfd6c-2faa-45a9-a5eb-3fc2aac37bcd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:05.633920 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-457bfd6c-2faa-45a9-a5eb-3fc2aac37bcd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:05.638146 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-457bfd6c-2faa-45a9-a5eb-3fc2aac37bcd None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:05.655227 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 282/563] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:12:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:05.662032 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-457bfd6c-2faa-45a9-a5eb-3fc2aac37bcd None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:05.669574 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e2ffbfa-70f8-46c5-a541-13cf7d3ff596 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:05.718640 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563cec3f-fe57-403a-8ef5-4fb0d78acd76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:05.719658 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563cec3f-fe57-403a-8ef5-4fb0d78acd76 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:05.720143 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563cec3f-fe57-403a-8ef5-4fb0d78acd76 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:05.720688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563cec3f-fe57-403a-8ef5-4fb0d78acd76 None admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:05.721722 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-563cec3f-fe57-403a-8ef5-4fb0d78acd76 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:05.722904 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-563cec3f-fe57-403a-8ef5-4fb0d78acd76 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:05.724653 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-563cec3f-fe57-403a-8ef5-4fb0d78acd76 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:05.743927 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 282/564] 10.222.0.42 () {56 vars in 1133 bytes} [Thu Oct 27 01:12:05 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:05.745122 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563cec3f-fe57-403a-8ef5-4fb0d78acd76 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:05.778620 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-457bfd6c-2faa-45a9-a5eb-3fc2aac37bcd None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:05.825709 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948b4d32-c6bd-4e1c-9781-30bdf4929739 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:05.826322 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948b4d32-c6bd-4e1c-9781-30bdf4929739 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:05.826602 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948b4d32-c6bd-4e1c-9781-30bdf4929739 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:05.826875 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948b4d32-c6bd-4e1c-9781-30bdf4929739 None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:05.827577 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948b4d32-c6bd-4e1c-9781-30bdf4929739 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:05.827885 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948b4d32-c6bd-4e1c-9781-30bdf4929739 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:05.829351 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948b4d32-c6bd-4e1c-9781-30bdf4929739 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:05.847204 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 283/565] 10.222.0.42 () {56 vars in 1148 bytes} [Thu Oct 27 01:12:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:05.848099 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948b4d32-c6bd-4e1c-9781-30bdf4929739 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:05.857751 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563cec3f-fe57-403a-8ef5-4fb0d78acd76 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:05.883699 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:05.898380 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93dad1b-8a2c-41f3-a1f8-827904c90b00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:05.898683 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:05.899431 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93dad1b-8a2c-41f3-a1f8-827904c90b00 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:05.900262 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93dad1b-8a2c-41f3-a1f8-827904c90b00 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:05.900657 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93dad1b-8a2c-41f3-a1f8-827904c90b00 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:05.910511 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93dad1b-8a2c-41f3-a1f8-827904c90b00 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:05.912735 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93dad1b-8a2c-41f3-a1f8-827904c90b00 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:05.929094 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-f93dad1b-8a2c-41f3-a1f8-827904c90b00 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:12:05.929094 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:05.929094 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:05.929094 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:05.929094 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:05.929094 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:05.929094 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:05.929094 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:05.929094 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:05.929094 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:05.929094 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:05.929094 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:05.929094 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 27 01:12:05.929094 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 27 01:12:05.929094 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 27 01:12:05.930257 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 27 01:12:05.930257 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:05.930257 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:05.930257 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:05.930257 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:05.930257 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:05.930257 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:05.930257 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:05.930257 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:05.930257 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:05.930257 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:05.930257 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:05.930257 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:05.930257 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:05.930257 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:05.930257 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:05.930257 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:05.931648 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 27 01:12:05.931648 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 27 01:12:05.931648 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 27 01:12:05.931648 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 27 01:12:05.931648 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 27 01:12:05.931648 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 27 01:12:05.931648 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:12:05.931648 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:05.931648 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 283/566] 10.222.0.42 () {56 vars in 1111 bytes} [Thu Oct 27 01:12:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:05.936869 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93dad1b-8a2c-41f3-a1f8-827904c90b00 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:05.976548 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948b4d32-c6bd-4e1c-9781-30bdf4929739 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:06.013673 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109b1ca0-dbe6-4232-a5cb-9b683ebefd16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:06.014521 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109b1ca0-dbe6-4232-a5cb-9b683ebefd16 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:06.014895 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109b1ca0-dbe6-4232-a5cb-9b683ebefd16 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:06.015270 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109b1ca0-dbe6-4232-a5cb-9b683ebefd16 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:06.063786 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93dad1b-8a2c-41f3-a1f8-827904c90b00 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:06.090551 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109b1ca0-dbe6-4232-a5cb-9b683ebefd16 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:06.092148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109b1ca0-dbe6-4232-a5cb-9b683ebefd16 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:06.109747 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b7ff8e-7ab4-402f-8e7c-f7e00722ae7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:06.110415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b7ff8e-7ab4-402f-8e7c-f7e00722ae7c None admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:06.110693 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b7ff8e-7ab4-402f-8e7c-f7e00722ae7c None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:06.110973 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b7ff8e-7ab4-402f-8e7c-f7e00722ae7c None admin] PATH_INFO: `/v3/projects/23aad848d47c4d27ad4c138ff869e40f/users/77f2cf9e0c3541f9bccfb7e213d88609/roles/414286927fdd44688107ff5e10235acf` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:06.113705 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21b7ff8e-7ab4-402f-8e7c-f7e00722ae7c None admin] RBAC: Authorizing `identity:create_grant(project_id=23aad848d47c4d27ad4c138ff869e40f, user_id=77f2cf9e0c3541f9bccfb7e213d88609, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:06.115215 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21b7ff8e-7ab4-402f-8e7c-f7e00722ae7c None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:06.138641 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 284/567] 10.222.0.42 () {58 vars in 1439 bytes} [Thu Oct 27 01:12:05 2022] PUT /identity/v3/projects/23aad848d47c4d27ad4c138ff869e40f/users/77f2cf9e0c3541f9bccfb7e213d88609/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:12:06.139412 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b7ff8e-7ab4-402f-8e7c-f7e00722ae7c None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:06.149500 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:06.201719 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:06.216112 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 284/568] 10.222.0.42 () {58 vars in 1282 bytes} [Thu Oct 27 01:12:05 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 27 01:12:06.222850 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109b1ca0-dbe6-4232-a5cb-9b683ebefd16 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:06.274383 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b7ff8e-7ab4-402f-8e7c-f7e00722ae7c None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:06.313013 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a52ced3-4191-4106-a07f-3b27197588f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:06.313959 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a52ced3-4191-4106-a07f-3b27197588f9 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:06.314383 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a52ced3-4191-4106-a07f-3b27197588f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:06.314801 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a52ced3-4191-4106-a07f-3b27197588f9 None admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:06.316272 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a52ced3-4191-4106-a07f-3b27197588f9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:06.316888 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a52ced3-4191-4106-a07f-3b27197588f9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:06.318644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a52ced3-4191-4106-a07f-3b27197588f9 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:06.333202 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 285/569] 10.222.0.42 () {56 vars in 1148 bytes} [Thu Oct 27 01:12:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:06.346116 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a52ced3-4191-4106-a07f-3b27197588f9 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:06.415824 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109b1ca0-dbe6-4232-a5cb-9b683ebefd16 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:06.460544 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a2fd7e9-b3ba-47ba-88a9-7ccbba3fab14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:06.461745 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2fd7e9-b3ba-47ba-88a9-7ccbba3fab14 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:06.462129 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2fd7e9-b3ba-47ba-88a9-7ccbba3fab14 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:06.462501 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2fd7e9-b3ba-47ba-88a9-7ccbba3fab14 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:06.475255 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a2fd7e9-b3ba-47ba-88a9-7ccbba3fab14 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:06.476452 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a52ced3-4191-4106-a07f-3b27197588f9 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:06.477011 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a2fd7e9-b3ba-47ba-88a9-7ccbba3fab14 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:06.478165 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:06.486845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:06.498061 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-6a2fd7e9-b3ba-47ba-88a9-7ccbba3fab14 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:12:06.498061 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:06.498061 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:06.498061 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:06.498061 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:06.498061 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:06.498061 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:06.498061 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:06.498061 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:06.498061 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:06.498061 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:06.498061 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:06.498061 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:12:06.498061 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:12:06.498061 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:12:06.498061 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:12:06.499336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:06.499336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:06.499336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:06.499336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:06.499336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:06.499336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:06.499336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:06.499336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:06.499336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:06.499336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:06.499336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:06.499336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:06.499336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:06.499336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:06.499336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:06.499336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:06.499336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:12:06.500408 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:12:06.500408 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:12:06.500408 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:06.506385 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 285/570] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:12:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 281 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:06.506385 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b90adb1-339a-4a55-b991-f036bc9e64e8 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:06.506385 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b90adb1-339a-4a55-b991-f036bc9e64e8 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:06.506385 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b90adb1-339a-4a55-b991-f036bc9e64e8 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:06.506385 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 286/571] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:06.511101 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b90adb1-339a-4a55-b991-f036bc9e64e8 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:06.528852 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975f3c90-eb20-4ed1-9601-637dcc29cc38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:06.529735 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975f3c90-eb20-4ed1-9601-637dcc29cc38 None admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:06.530163 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975f3c90-eb20-4ed1-9601-637dcc29cc38 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:06.530584 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975f3c90-eb20-4ed1-9601-637dcc29cc38 None admin] PATH_INFO: `/v3/projects/23aad848d47c4d27ad4c138ff869e40f/users/7a3f8b5eb09f4ba29f4bd343bd91f6a2/roles/1b8f8dd9e8584053b048032db63254a4` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:06.551936 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-975f3c90-eb20-4ed1-9601-637dcc29cc38 None admin] RBAC: Authorizing `identity:create_grant(project_id=23aad848d47c4d27ad4c138ff869e40f, user_id=7a3f8b5eb09f4ba29f4bd343bd91f6a2, role_id=1b8f8dd9e8584053b048032db63254a4)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:06.553660 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-975f3c90-eb20-4ed1-9601-637dcc29cc38 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:06.571172 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 286/572] 10.222.0.42 () {58 vars in 1439 bytes} [Thu Oct 27 01:12:06 2022] PUT /identity/v3/projects/23aad848d47c4d27ad4c138ff869e40f/users/7a3f8b5eb09f4ba29f4bd343bd91f6a2/roles/1b8f8dd9e8584053b048032db63254a4 => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:12:06.576127 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb981042-3303-479f-b0da-746d6441f6ad None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:06.576127 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb981042-3303-479f-b0da-746d6441f6ad None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:06.576127 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb981042-3303-479f-b0da-746d6441f6ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:06.601455 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-bb981042-3303-479f-b0da-746d6441f6ad None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:06.604555 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-bb981042-3303-479f-b0da-746d6441f6ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:06.676088 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b90adb1-339a-4a55-b991-f036bc9e64e8 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:06.748324 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba3233da-fd9f-4396-b087-1dd22d702698 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:06.748941 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3233da-fd9f-4396-b087-1dd22d702698 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:06.749244 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3233da-fd9f-4396-b087-1dd22d702698 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:06.749539 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3233da-fd9f-4396-b087-1dd22d702698 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:06.750169 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba3233da-fd9f-4396-b087-1dd22d702698 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:06.750472 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba3233da-fd9f-4396-b087-1dd22d702698 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:06.751876 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba3233da-fd9f-4396-b087-1dd22d702698 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:06.755431 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 287/573] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:06 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:12:06.768630 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 287/574] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:12:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:06.774136 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb981042-3303-479f-b0da-746d6441f6ad None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:06.900663 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:06.924446 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb981042-3303-479f-b0da-746d6441f6ad None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:06.963459 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc02501-74d1-4d07-94d0-1259bbf5813f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:06.964018 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc02501-74d1-4d07-94d0-1259bbf5813f None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:06.964295 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc02501-74d1-4d07-94d0-1259bbf5813f None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:06.964566 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc02501-74d1-4d07-94d0-1259bbf5813f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:06.972583 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfc02501-74d1-4d07-94d0-1259bbf5813f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:06.974056 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfc02501-74d1-4d07-94d0-1259bbf5813f None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:06.986443 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-bfc02501-74d1-4d07-94d0-1259bbf5813f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:12:06.986443 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:06.986443 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:06.986443 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:06.986443 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:06.986443 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:06.986443 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:06.986443 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:06.986443 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:06.986443 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:06.986443 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:06.986443 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:06.986443 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 27 01:12:06.986443 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 27 01:12:06.986443 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 27 01:12:06.987422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 27 01:12:06.987422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:06.987422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:06.987422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:06.987422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:06.987422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:06.987422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:06.987422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:06.987422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:06.987422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:06.987422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:06.987422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:06.987422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:06.987422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:06.987422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:06.987422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:06.987422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:06.988817 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 27 01:12:06.988817 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 27 01:12:06.988817 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 27 01:12:06.988817 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 27 01:12:06.988817 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 27 01:12:06.988817 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 27 01:12:06.988817 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:12:06.988817 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:06.988817 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 288/575] 10.222.0.42 () {56 vars in 1111 bytes} [Thu Oct 27 01:12:06 2022] GET /identity/v3/projects/service => generated 88 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 27 01:12:06.996985 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba3233da-fd9f-4396-b087-1dd22d702698 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:07.095430 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:07.112223 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29845afd-9578-4e2a-997c-d4a62e1dadf3 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:07.113038 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29845afd-9578-4e2a-997c-d4a62e1dadf3 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:07.113353 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29845afd-9578-4e2a-997c-d4a62e1dadf3 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:07.116590 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 289/576] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:07.124417 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d7e00f-6bd7-4e5e-a8b7-c2a296a2ba53 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:07.124699 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d7e00f-6bd7-4e5e-a8b7-c2a296a2ba53 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:07.124970 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d7e00f-6bd7-4e5e-a8b7-c2a296a2ba53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:07.143013 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba3233da-fd9f-4396-b087-1dd22d702698 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:07.145352 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-59d7e00f-6bd7-4e5e-a8b7-c2a296a2ba53 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:07.147490 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-59d7e00f-6bd7-4e5e-a8b7-c2a296a2ba53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:07.184725 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f77d2c2-bfa9-4104-b8ed-a9022a879fdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:07.185385 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f77d2c2-bfa9-4104-b8ed-a9022a879fdd None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:07.185638 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f77d2c2-bfa9-4104-b8ed-a9022a879fdd None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:07.185912 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f77d2c2-bfa9-4104-b8ed-a9022a879fdd None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:07.186725 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f77d2c2-bfa9-4104-b8ed-a9022a879fdd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:07.187030 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f77d2c2-bfa9-4104-b8ed-a9022a879fdd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:07.188451 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f77d2c2-bfa9-4104-b8ed-a9022a879fdd None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:07.208580 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 288/577] 10.222.0.42 () {56 vars in 1148 bytes} [Thu Oct 27 01:12:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:07.208580 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f77d2c2-bfa9-4104-b8ed-a9022a879fdd None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:07.245318 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 290/578] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:07 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:12:07.246753 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59d7e00f-6bd7-4e5e-a8b7-c2a296a2ba53 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:07.382597 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:07.394664 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f77d2c2-bfa9-4104-b8ed-a9022a879fdd None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:07.430382 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:07.446792 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59d7e00f-6bd7-4e5e-a8b7-c2a296a2ba53 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:07.452510 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef6697e-c0d6-4c8e-9ad0-9883402460bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:07.454250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef6697e-c0d6-4c8e-9ad0-9883402460bd None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:07.454682 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef6697e-c0d6-4c8e-9ad0-9883402460bd None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:07.458565 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef6697e-c0d6-4c8e-9ad0-9883402460bd None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:07.460949 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef6697e-c0d6-4c8e-9ad0-9883402460bd None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:07.462599 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef6697e-c0d6-4c8e-9ad0-9883402460bd None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:07.481215 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-eef6697e-c0d6-4c8e-9ad0-9883402460bd None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 27 01:12:07.481215 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:07.481215 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 27 01:12:07.481215 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 27 01:12:07.481215 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 27 01:12:07.481215 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 27 01:12:07.481215 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 27 01:12:07.481215 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:07.481215 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 27 01:12:07.481215 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:07.481215 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 27 01:12:07.481215 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:07.481215 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:07.481215 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:07.481215 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:07.481215 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:07.481215 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:07.482494 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:07.482494 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:07.482494 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:07.482494 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:07.482494 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:07.482494 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:07.482494 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 27 01:12:07.482494 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 27 01:12:07.482494 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 27 01:12:07.482494 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 27 01:12:07.482494 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:07.482494 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:07.482494 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:07.482494 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:07.482494 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:07.482494 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:07.483442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:07.483442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:07.483442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:07.483442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:07.483442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:07.483442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:07.483442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:07.483442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:07.483442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:07.483442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:07.483442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 27 01:12:07.483442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:07.483442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 27 01:12:07.483442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:07.486154 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 289/579] 10.222.0.42 () {56 vars in 1129 bytes} [Thu Oct 27 01:12:07 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 282 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:07.492901 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef6697e-c0d6-4c8e-9ad0-9883402460bd None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:07.518480 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49cc11ea-cd25-442e-87f6-b4896438ee17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:07.519842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49cc11ea-cd25-442e-87f6-b4896438ee17 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:07.520443 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49cc11ea-cd25-442e-87f6-b4896438ee17 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:07.522448 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49cc11ea-cd25-442e-87f6-b4896438ee17 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:07.524113 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49cc11ea-cd25-442e-87f6-b4896438ee17 None admin] RBAC: Adding query filter params (role.id=414286927fdd44688107ff5e10235acf, scope.project.id=23aad848d47c4d27ad4c138ff869e40f, user.id=51033d56f85f4282a49ded6338440677) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:07.524604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49cc11ea-cd25-442e-87f6-b4896438ee17 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:07.527259 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49cc11ea-cd25-442e-87f6-b4896438ee17 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:07.541122 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 291/580] 10.222.0.42 () {56 vars in 1374 bytes} [Thu Oct 27 01:12:07 2022] GET /identity/v3/role_assignments?user.id=51033d56f85f4282a49ded6338440677&scope.project.id=23aad848d47c4d27ad4c138ff869e40f&role.id=414286927fdd44688107ff5e10235acf => generated 608 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:07.578336 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49cc11ea-cd25-442e-87f6-b4896438ee17 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:07.685570 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef6697e-c0d6-4c8e-9ad0-9883402460bd None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:07.759603 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78fca3e9-b6bd-4903-837c-6c70dc719acc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:07.760221 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78fca3e9-b6bd-4903-837c-6c70dc719acc None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:07.760499 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78fca3e9-b6bd-4903-837c-6c70dc719acc None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:07.760770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78fca3e9-b6bd-4903-837c-6c70dc719acc None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:07.761447 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78fca3e9-b6bd-4903-837c-6c70dc719acc None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:07.761751 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78fca3e9-b6bd-4903-837c-6c70dc719acc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:07.763124 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78fca3e9-b6bd-4903-837c-6c70dc719acc None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:07.774381 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 290/581] 10.222.0.42 () {56 vars in 1124 bytes} [Thu Oct 27 01:12:07 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:12:07.776734 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49cc11ea-cd25-442e-87f6-b4896438ee17 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:07.780039 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78fca3e9-b6bd-4903-837c-6c70dc719acc None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:07.825814 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a7adcc-ebce-46ef-8cc6-4de18892e8ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:07.826447 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a7adcc-ebce-46ef-8cc6-4de18892e8ab None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:07.826727 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a7adcc-ebce-46ef-8cc6-4de18892e8ab None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:07.826999 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a7adcc-ebce-46ef-8cc6-4de18892e8ab None admin] PATH_INFO: `/v3/services/network` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:07.827589 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a7adcc-ebce-46ef-8cc6-4de18892e8ab None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:07.829742 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a7adcc-ebce-46ef-8cc6-4de18892e8ab None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:07.832598 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:07.840498 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-72a7adcc-ebce-46ef-8cc6-4de18892e8ab None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 27 01:12:07.840498 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:07.840498 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 27 01:12:07.840498 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 27 01:12:07.840498 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 27 01:12:07.840498 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 27 01:12:07.840498 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 27 01:12:07.840498 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:07.840498 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 27 01:12:07.840498 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:07.840498 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 27 01:12:07.840498 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:07.840498 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:07.840498 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:07.840498 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:07.840498 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:07.840498 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:07.841615 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:07.841615 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:07.841615 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:07.841615 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:07.841615 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:07.841615 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:07.841615 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 27 01:12:07.841615 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 27 01:12:07.841615 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 27 01:12:07.841615 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 27 01:12:07.841615 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:07.841615 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:07.841615 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:07.841615 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:07.841615 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:07.841615 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:07.842537 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:07.842537 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:07.842537 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:07.842537 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:07.842537 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:07.842537 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:07.842537 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:07.842537 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:07.842537 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:07.842537 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:07.842537 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 27 01:12:07.842537 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:07.842537 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 27 01:12:07.842537 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:07.845391 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 292/582] 10.222.0.42 () {56 vars in 1111 bytes} [Thu Oct 27 01:12:07 2022] GET /identity/v3/services/network => generated 88 bytes in 267 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 27 01:12:07.847819 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:07.853510 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a7adcc-ebce-46ef-8cc6-4de18892e8ab None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:07.923565 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78fca3e9-b6bd-4903-837c-6c70dc719acc None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:07.967834 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd180e88-a44b-48d5-8564-c96b62619e07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:07.968439 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd180e88-a44b-48d5-8564-c96b62619e07 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:07.968715 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd180e88-a44b-48d5-8564-c96b62619e07 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:07.968988 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd180e88-a44b-48d5-8564-c96b62619e07 None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:07.969668 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd180e88-a44b-48d5-8564-c96b62619e07 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:07.969969 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd180e88-a44b-48d5-8564-c96b62619e07 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:07.971593 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd180e88-a44b-48d5-8564-c96b62619e07 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:07.978074 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 291/583] 10.222.0.42 () {56 vars in 1124 bytes} [Thu Oct 27 01:12:07 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:12:07.983896 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd180e88-a44b-48d5-8564-c96b62619e07 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:08.004701 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a7adcc-ebce-46ef-8cc6-4de18892e8ab None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:08.043070 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d252cf6c-8cb4-4951-a8a3-59e46cb0f896 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:08.043969 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d252cf6c-8cb4-4951-a8a3-59e46cb0f896 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:08.044499 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d252cf6c-8cb4-4951-a8a3-59e46cb0f896 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:08.044970 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d252cf6c-8cb4-4951-a8a3-59e46cb0f896 None admin] PATH_INFO: `/v3/services` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:08.045756 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d252cf6c-8cb4-4951-a8a3-59e46cb0f896 None admin] RBAC: Adding query filter params (name=network) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:08.046208 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d252cf6c-8cb4-4951-a8a3-59e46cb0f896 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:08.047714 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d252cf6c-8cb4-4951-a8a3-59e46cb0f896 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:08.053630 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 293/584] 10.222.0.42 () {56 vars in 1112 bytes} [Thu Oct 27 01:12:07 2022] GET /identity/v3/services?name=network => generated 124 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:08.059088 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d252cf6c-8cb4-4951-a8a3-59e46cb0f896 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:08.088347 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd180e88-a44b-48d5-8564-c96b62619e07 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:08.099852 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:08.133869 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd79664a-cdad-4005-bb52-81207a1a4f94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:08.135043 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd79664a-cdad-4005-bb52-81207a1a4f94 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:08.135707 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd79664a-cdad-4005-bb52-81207a1a4f94 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:08.136135 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd79664a-cdad-4005-bb52-81207a1a4f94 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:08.137664 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd79664a-cdad-4005-bb52-81207a1a4f94 None admin] RBAC: Adding query filter params (interface=public, service_id=790cbeec745c4670a0b7a505c5ea24f0, region_id=RegionOne) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:08.138130 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd79664a-cdad-4005-bb52-81207a1a4f94 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:08.139888 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:08.141685 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd79664a-cdad-4005-bb52-81207a1a4f94 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:08.154950 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 292/585] 10.222.0.42 () {56 vars in 1251 bytes} [Thu Oct 27 01:12:07 2022] GET /identity/v3/endpoints?service_id=790cbeec745c4670a0b7a505c5ea24f0&interface=public®ion_id=RegionOne => generated 194 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:12:08.163013 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd79664a-cdad-4005-bb52-81207a1a4f94 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:08.186704 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d252cf6c-8cb4-4951-a8a3-59e46cb0f896 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:08.270535 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f79a718-bc4d-475f-9555-d4f1fc24e060 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:08.271155 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f79a718-bc4d-475f-9555-d4f1fc24e060 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:08.271431 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f79a718-bc4d-475f-9555-d4f1fc24e060 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:08.271701 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f79a718-bc4d-475f-9555-d4f1fc24e060 None admin] PATH_INFO: `/v3/services` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:08.272333 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f79a718-bc4d-475f-9555-d4f1fc24e060 None admin] RBAC: Adding query filter params (type=network) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:08.272632 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f79a718-bc4d-475f-9555-d4f1fc24e060 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:08.274074 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f79a718-bc4d-475f-9555-d4f1fc24e060 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:08.287871 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd79664a-cdad-4005-bb52-81207a1a4f94 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:08.289763 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 294/586] 10.222.0.42 () {56 vars in 1112 bytes} [Thu Oct 27 01:12:08 2022] GET /identity/v3/services?type=network => generated 350 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:12:08.295707 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f79a718-bc4d-475f-9555-d4f1fc24e060 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:08.337919 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844b5bc5-b15b-40e5-a711-e3e4865bc58a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:08.339189 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844b5bc5-b15b-40e5-a711-e3e4865bc58a None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:08.339189 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844b5bc5-b15b-40e5-a711-e3e4865bc58a None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:08.339189 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844b5bc5-b15b-40e5-a711-e3e4865bc58a None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:08.339650 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-844b5bc5-b15b-40e5-a711-e3e4865bc58a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:08.341024 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-844b5bc5-b15b-40e5-a711-e3e4865bc58a None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:08.346681 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 293/587] 10.222.0.42 () {56 vars in 1087 bytes} [Thu Oct 27 01:12:08 2022] GET /identity/v3/services => generated 772 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:08.415761 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f79a718-bc4d-475f-9555-d4f1fc24e060 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:08.445774 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:08.454062 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7477dc1-f231-421d-886f-ff5772d20f72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:08.455168 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7477dc1-f231-421d-886f-ff5772d20f72 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:08.455712 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7477dc1-f231-421d-886f-ff5772d20f72 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:08.456387 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7477dc1-f231-421d-886f-ff5772d20f72 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:08.457352 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7477dc1-f231-421d-886f-ff5772d20f72 None admin] RBAC: Adding query filter params (interface=public, service_id=1ae802467bcd4250948eaaca79251a06, region_id=RegionOne) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:08.457562 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7477dc1-f231-421d-886f-ff5772d20f72 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:08.459492 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7477dc1-f231-421d-886f-ff5772d20f72 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:08.468809 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 295/588] 10.222.0.42 () {56 vars in 1251 bytes} [Thu Oct 27 01:12:08 2022] GET /identity/v3/endpoints?service_id=1ae802467bcd4250948eaaca79251a06&interface=public®ion_id=RegionOne => generated 194 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:12:08.474997 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844b5bc5-b15b-40e5-a711-e3e4865bc58a None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:08.539945 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:08.619152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844b5bc5-b15b-40e5-a711-e3e4865bc58a None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:08.658806 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e4143b-3777-4df5-8960-2a0998e1e42d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:08.659519 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e4143b-3777-4df5-8960-2a0998e1e42d None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:08.660447 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e4143b-3777-4df5-8960-2a0998e1e42d None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:08.660728 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e4143b-3777-4df5-8960-2a0998e1e42d None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:08.661403 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e4143b-3777-4df5-8960-2a0998e1e42d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:08.662801 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e4143b-3777-4df5-8960-2a0998e1e42d None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:08.668913 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 294/589] 10.222.0.42 () {56 vars in 1087 bytes} [Thu Oct 27 01:12:08 2022] GET /identity/v3/services => generated 772 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:08.776491 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c345d44e-0417-4511-8477-d1df8a17c2eb None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:08.777463 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c345d44e-0417-4511-8477-d1df8a17c2eb None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:08.777918 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c345d44e-0417-4511-8477-d1df8a17c2eb None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:08.779733 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 296/590] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:08.786292 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cebc372-2783-463f-b750-6986e673245c None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:08.786626 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cebc372-2783-463f-b750-6986e673245c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:08.786837 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cebc372-2783-463f-b750-6986e673245c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:08.814356 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-2cebc372-2783-463f-b750-6986e673245c None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:08.815507 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-2cebc372-2783-463f-b750-6986e673245c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:08.912191 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 295/591] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:08 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 27 01:12:09.093982 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381d7111-5262-4062-91d9-97c555e9726a None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:09.094324 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381d7111-5262-4062-91d9-97c555e9726a None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:09.094539 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381d7111-5262-4062-91d9-97c555e9726a None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:09.096138 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 297/592] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:09.102426 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80122d0-ce1c-44ba-aa56-c7bc58571935 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:09.102705 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80122d0-ce1c-44ba-aa56-c7bc58571935 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:09.102978 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80122d0-ce1c-44ba-aa56-c7bc58571935 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:09.126177 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-e80122d0-ce1c-44ba-aa56-c7bc58571935 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:09.127400 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e80122d0-ce1c-44ba-aa56-c7bc58571935 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:09.165869 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:09.220395 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-381d7111-5262-4062-91d9-97c555e9726a None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:09.221009 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 296/593] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:09 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:12:09.380670 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-381d7111-5262-4062-91d9-97c555e9726a None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:09.425813 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b801fc48-0b4e-4384-aa9c-4ac671573b5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:09.426404 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b801fc48-0b4e-4384-aa9c-4ac671573b5c None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:09.426683 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b801fc48-0b4e-4384-aa9c-4ac671573b5c None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:09.426954 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b801fc48-0b4e-4384-aa9c-4ac671573b5c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:09.449894 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b801fc48-0b4e-4384-aa9c-4ac671573b5c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:12:09.454708 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-b801fc48-0b4e-4384-aa9c-4ac671573b5c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:12:09.454708 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:09.454708 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:09.454708 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:09.454708 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:09.454708 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:09.454708 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:09.454708 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:09.454708 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:09.454708 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:09.454708 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:09.454708 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:09.454708 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:12:09.454708 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:12:09.454708 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:12:09.454708 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:12:09.455630 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:12:09.455630 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:12:09.455630 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:12:09.455630 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:12:09.455630 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:09.455630 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:09.455630 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:09.455630 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:09.455630 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:09.455630 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:09.455630 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:09.455630 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:09.455630 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:09.455630 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:09.455630 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:09.455630 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:09.457085 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:09.457085 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:09.457085 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:09.457085 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:09.457085 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:12:09.457085 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:12:09.457085 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:12:09.457085 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:12:09.457085 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:12:09.457085 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:12:09.457085 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:12:09.457085 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:09.457085 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 298/594] 10.222.0.42 () {56 vars in 1096 bytes} [Thu Oct 27 01:12:09 2022] GET /identity/v3/roles/admin => generated 83 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:09.462621 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80122d0-ce1c-44ba-aa56-c7bc58571935 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:09.566116 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b801fc48-0b4e-4384-aa9c-4ac671573b5c None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:09.601181 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80122d0-ce1c-44ba-aa56-c7bc58571935 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:09.662069 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:09.662298 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0875945-747b-4fef-b96c-5e2d5c8b4d49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:09.662864 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0875945-747b-4fef-b96c-5e2d5c8b4d49 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:09.663138 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0875945-747b-4fef-b96c-5e2d5c8b4d49 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:09.663413 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0875945-747b-4fef-b96c-5e2d5c8b4d49 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:09.664079 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0875945-747b-4fef-b96c-5e2d5c8b4d49 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:09.664387 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0875945-747b-4fef-b96c-5e2d5c8b4d49 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:09.665786 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0875945-747b-4fef-b96c-5e2d5c8b4d49 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:09.688708 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 297/595] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:12:09 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:09.693221 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0875945-747b-4fef-b96c-5e2d5c8b4d49 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:09.740422 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:09.753829 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b801fc48-0b4e-4384-aa9c-4ac671573b5c None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:09.822660 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6b3450-b3be-4d30-8adb-1fc8bb79c57c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:09.823435 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6b3450-b3be-4d30-8adb-1fc8bb79c57c None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:09.823648 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6b3450-b3be-4d30-8adb-1fc8bb79c57c None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:09.823920 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6b3450-b3be-4d30-8adb-1fc8bb79c57c None admin] PATH_INFO: `/v3/roles/service` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:09.841231 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6b3450-b3be-4d30-8adb-1fc8bb79c57c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:12:09.849492 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-fd6b3450-b3be-4d30-8adb-1fc8bb79c57c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 27 01:12:09.849492 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:09.849492 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:09.849492 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:09.849492 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:09.849492 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:09.849492 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:09.849492 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:09.849492 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:09.849492 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:09.849492 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:09.849492 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:09.849492 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:12:09.849492 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:12:09.849492 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:12:09.849492 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:12:09.850497 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:12:09.850497 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:12:09.850497 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:12:09.850497 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:12:09.850497 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:09.850497 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:09.850497 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:09.850497 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:09.850497 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:09.850497 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:09.850497 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:09.850497 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:09.850497 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:09.850497 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:09.850497 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:09.850497 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:09.851387 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:09.851387 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:09.851387 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:09.851387 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:09.851387 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:12:09.851387 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:12:09.851387 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:12:09.851387 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:12:09.851387 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:12:09.851387 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:12:09.851387 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 27 01:12:09.851387 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:09.857182 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 299/596] 10.222.0.42 () {56 vars in 1102 bytes} [Thu Oct 27 01:12:09 2022] GET /identity/v3/roles/service => generated 85 bytes in 286 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 27 01:12:09.859538 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6b3450-b3be-4d30-8adb-1fc8bb79c57c None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:09.888359 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0875945-747b-4fef-b96c-5e2d5c8b4d49 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:09.953650 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200de8a5-9967-4af5-98de-f67d71d804e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:09.954202 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200de8a5-9967-4af5-98de-f67d71d804e6 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:09.954473 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200de8a5-9967-4af5-98de-f67d71d804e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:09.954735 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200de8a5-9967-4af5-98de-f67d71d804e6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:09.977883 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-200de8a5-9967-4af5-98de-f67d71d804e6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:09.979323 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-200de8a5-9967-4af5-98de-f67d71d804e6 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:10.000040 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-200de8a5-9967-4af5-98de-f67d71d804e6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:12:10.000040 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:10.000040 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:10.000040 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:10.000040 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:10.000040 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:10.000040 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:10.000040 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:10.000040 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:10.000040 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:10.000040 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:10.000040 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:10.000040 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:12:10.000040 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:12:10.000040 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:12:10.000040 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:12:10.001080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:10.001080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:10.001080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:10.001080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:10.001080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:10.001080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:10.001080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:10.001080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:10.001080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:10.001080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:10.001080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:10.001080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:10.001080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:10.001080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:10.001080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:10.001080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:10.001080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:12:10.001980 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:12:10.001980 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:12:10.001980 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:10.001980 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 298/597] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:12:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 308 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:10.008856 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200de8a5-9967-4af5-98de-f67d71d804e6 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:10.040934 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6b3450-b3be-4d30-8adb-1fc8bb79c57c None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:10.055170 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:10.078977 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1b1b4a-f914-4274-81ab-848b34dae796 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:10.079634 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1b1b4a-f914-4274-81ab-848b34dae796 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:10.079910 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1b1b4a-f914-4274-81ab-848b34dae796 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:10.080233 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1b1b4a-f914-4274-81ab-848b34dae796 None admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:10.080896 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd1b1b4a-f914-4274-81ab-848b34dae796 None admin] RBAC: Adding query filter params (name=service) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:10.081389 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd1b1b4a-f914-4274-81ab-848b34dae796 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:10.083516 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd1b1b4a-f914-4274-81ab-848b34dae796 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:10.092351 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 300/598] 10.222.0.42 () {56 vars in 1103 bytes} [Thu Oct 27 01:12:09 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:10.097886 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1b1b4a-f914-4274-81ab-848b34dae796 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:10.126385 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:10.165212 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200de8a5-9967-4af5-98de-f67d71d804e6 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:10.211736 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1b1b4a-f914-4274-81ab-848b34dae796 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:10.232608 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc26f8be-5dba-4197-86b6-b87263ab88ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:10.233392 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc26f8be-5dba-4197-86b6-b87263ab88ff None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:10.233674 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc26f8be-5dba-4197-86b6-b87263ab88ff None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:10.233945 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc26f8be-5dba-4197-86b6-b87263ab88ff None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:10.234830 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc26f8be-5dba-4197-86b6-b87263ab88ff None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:10.235138 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc26f8be-5dba-4197-86b6-b87263ab88ff None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:10.237260 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc26f8be-5dba-4197-86b6-b87263ab88ff None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:10.262280 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4dbfe9-b3f6-4f9f-add1-4d101c8dac62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:10.262933 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4dbfe9-b3f6-4f9f-add1-4d101c8dac62 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:10.263209 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4dbfe9-b3f6-4f9f-add1-4d101c8dac62 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:10.263477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4dbfe9-b3f6-4f9f-add1-4d101c8dac62 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:10.273157 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 299/599] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:12:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:12:10.275699 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc26f8be-5dba-4197-86b6-b87263ab88ff None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:10.289842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c4dbfe9-b3f6-4f9f-add1-4d101c8dac62 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:10.291327 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c4dbfe9-b3f6-4f9f-add1-4d101c8dac62 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:10.319414 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-2c4dbfe9-b3f6-4f9f-add1-4d101c8dac62 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:12:10.319414 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:10.319414 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:10.319414 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:10.319414 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:10.319414 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:10.319414 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:10.319414 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:10.319414 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:10.319414 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:10.319414 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:10.319414 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:10.319414 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:12:10.319414 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:12:10.319414 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:12:10.319414 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:12:10.320407 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:10.320407 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:10.320407 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:10.320407 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:10.320407 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:10.320407 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:10.320407 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:10.320407 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:10.320407 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:10.320407 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:10.320407 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:10.320407 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:10.320407 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:10.320407 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:10.320407 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:10.320407 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:10.320407 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:12:10.321307 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:12:10.321307 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:12:10.321307 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:10.322042 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 301/600] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:12:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:10.327846 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4dbfe9-b3f6-4f9f-add1-4d101c8dac62 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:10.384819 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:10.440104 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc26f8be-5dba-4197-86b6-b87263ab88ff None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:10.478922 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4dbfe9-b3f6-4f9f-add1-4d101c8dac62 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:10.479109 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d97930-113c-43d9-884a-8d25200b8d8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:10.479472 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d97930-113c-43d9-884a-8d25200b8d8c None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:10.479750 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d97930-113c-43d9-884a-8d25200b8d8c None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:10.480022 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d97930-113c-43d9-884a-8d25200b8d8c None admin] PATH_INFO: `/v3/users/swift` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:10.482939 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:10.489380 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60d97930-113c-43d9-884a-8d25200b8d8c None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:10.491003 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60d97930-113c-43d9-884a-8d25200b8d8c None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:10.504797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-60d97930-113c-43d9-884a-8d25200b8d8c None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 27 01:12:10.504797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:10.504797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:10.504797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:10.504797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:10.504797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:10.504797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:10.504797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:10.504797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:10.504797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:10.504797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:10.504797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:10.504797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 27 01:12:10.504797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 27 01:12:10.504797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 27 01:12:10.504797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 27 01:12:10.505811 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:10.505811 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:10.505811 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 27 01:12:10.505811 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:12:10.505811 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 27 01:12:10.505811 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:12:10.505811 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:10.505811 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:10.505811 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:10.505811 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:10.505811 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:10.505811 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:10.505811 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:10.505811 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:10.505811 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:10.505811 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:10.505811 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:10.505811 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:10.506760 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:10.506760 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:10.506760 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 27 01:12:10.506760 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 27 01:12:10.506760 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 27 01:12:10.506760 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 27 01:12:10.506760 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 27 01:12:10.506760 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 27 01:12:10.506760 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 27 01:12:10.506760 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:10.506760 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 300/601] 10.222.0.42 () {56 vars in 1096 bytes} [Thu Oct 27 01:12:10 2022] GET /identity/v3/users/swift => generated 83 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:10.507539 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e5a5be-a887-4c57-ad28-1b095f7c0eac None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:10.507967 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e5a5be-a887-4c57-ad28-1b095f7c0eac None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:10.508382 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e5a5be-a887-4c57-ad28-1b095f7c0eac None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:10.510634 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 301/602] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:10.511668 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e5a5be-a887-4c57-ad28-1b095f7c0eac None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:10.531073 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c3cdfe-fd29-4cd6-ac67-007d8b4cda95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:10.532039 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c3cdfe-fd29-4cd6-ac67-007d8b4cda95 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:10.532531 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c3cdfe-fd29-4cd6-ac67-007d8b4cda95 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:10.532954 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c3cdfe-fd29-4cd6-ac67-007d8b4cda95 None admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:10.533891 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2c3cdfe-fd29-4cd6-ac67-007d8b4cda95 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:10.534891 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2c3cdfe-fd29-4cd6-ac67-007d8b4cda95 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:10.536541 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2c3cdfe-fd29-4cd6-ac67-007d8b4cda95 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:10.550015 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 302/603] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:12:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:10.552259 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4761edbe-4796-4bd2-ab97-c7c0937a39bf None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:10.552749 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4761edbe-4796-4bd2-ab97-c7c0937a39bf None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:10.553273 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4761edbe-4796-4bd2-ab97-c7c0937a39bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:10.573931 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-4761edbe-4796-4bd2-ab97-c7c0937a39bf None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:10.575514 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-4761edbe-4796-4bd2-ab97-c7c0937a39bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:10.640907 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e5a5be-a887-4c57-ad28-1b095f7c0eac None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:10.649840 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 303/604] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:10 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:12:10.650933 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4761edbe-4796-4bd2-ab97-c7c0937a39bf None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:10.686814 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:10.707545 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa4b4a5-cddc-4411-9473-eb14a01d6016 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:10.709143 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa4b4a5-cddc-4411-9473-eb14a01d6016 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:10.710896 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa4b4a5-cddc-4411-9473-eb14a01d6016 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:10.712573 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa4b4a5-cddc-4411-9473-eb14a01d6016 None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:10.713547 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa4b4a5-cddc-4411-9473-eb14a01d6016 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:10.714962 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa4b4a5-cddc-4411-9473-eb14a01d6016 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:10.717740 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa4b4a5-cddc-4411-9473-eb14a01d6016 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:10.746711 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 302/605] 10.222.0.42 () {56 vars in 1135 bytes} [Thu Oct 27 01:12:10 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:10.755245 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa4b4a5-cddc-4411-9473-eb14a01d6016 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:10.866301 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4761edbe-4796-4bd2-ab97-c7c0937a39bf None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:10.881417 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:10.941610 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa4b4a5-cddc-4411-9473-eb14a01d6016 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:10.965793 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c283ee-eac4-41a8-99e4-96cf406d9d6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:10.966448 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c283ee-eac4-41a8-99e4-96cf406d9d6c None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:10.966728 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c283ee-eac4-41a8-99e4-96cf406d9d6c None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:10.967002 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c283ee-eac4-41a8-99e4-96cf406d9d6c None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:10.980408 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c283ee-eac4-41a8-99e4-96cf406d9d6c None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:10.981907 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c283ee-eac4-41a8-99e4-96cf406d9d6c None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:10.986055 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e35da9-bcf7-441b-966c-e1ab5318c199 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:10.986673 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e35da9-bcf7-441b-966c-e1ab5318c199 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:10.986951 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e35da9-bcf7-441b-966c-e1ab5318c199 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:10.987222 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e35da9-bcf7-441b-966c-e1ab5318c199 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:11.000664 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-13c283ee-eac4-41a8-99e4-96cf406d9d6c None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 27 01:12:11.000664 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:11.000664 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:11.000664 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:11.000664 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:11.000664 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:11.000664 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:11.000664 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:11.000664 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:11.000664 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:11.000664 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:11.000664 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:11.000664 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 27 01:12:11.000664 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 27 01:12:11.000664 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 27 01:12:11.000664 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 27 01:12:11.001678 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:11.001678 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:11.001678 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 27 01:12:11.001678 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:12:11.001678 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 27 01:12:11.001678 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:12:11.001678 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:11.001678 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:11.001678 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:11.001678 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:11.001678 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:11.001678 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:11.001678 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:11.001678 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:11.001678 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:11.001678 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:11.001678 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:11.001678 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:11.002632 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:11.002632 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:11.002632 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 27 01:12:11.002632 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 27 01:12:11.002632 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 27 01:12:11.002632 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 27 01:12:11.002632 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 27 01:12:11.002632 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 27 01:12:11.002632 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 27 01:12:11.002632 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:11.002632 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 304/606] 10.222.0.42 () {56 vars in 1117 bytes} [Thu Oct 27 01:12:10 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 351 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:11.003302 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c283ee-eac4-41a8-99e4-96cf406d9d6c None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:11.073479 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e35da9-bcf7-441b-966c-e1ab5318c199 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:11.076242 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e35da9-bcf7-441b-966c-e1ab5318c199 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:11.091972 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:11.121909 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c283ee-eac4-41a8-99e4-96cf406d9d6c None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:11.148823 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:11.157121 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4c0af5-1857-40f3-821f-4b08710d92d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:11.157751 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4c0af5-1857-40f3-821f-4b08710d92d2 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:11.158021 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4c0af5-1857-40f3-821f-4b08710d92d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:11.158333 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4c0af5-1857-40f3-821f-4b08710d92d2 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:11.158856 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f4c0af5-1857-40f3-821f-4b08710d92d2 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:11.161509 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f4c0af5-1857-40f3-821f-4b08710d92d2 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:11.171172 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-2f4c0af5-1857-40f3-821f-4b08710d92d2 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 27 01:12:11.171172 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:11.171172 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 27 01:12:11.171172 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 27 01:12:11.171172 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 27 01:12:11.171172 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 27 01:12:11.171172 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 27 01:12:11.171172 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:11.171172 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 27 01:12:11.171172 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:11.171172 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 27 01:12:11.171172 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:11.171172 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:11.171172 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:11.171172 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:11.171172 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:11.172416 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:11.172416 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:11.172416 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:11.172416 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:11.172416 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:11.172416 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:11.172416 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:11.172416 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 27 01:12:11.172416 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 27 01:12:11.172416 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 27 01:12:11.172416 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 27 01:12:11.172416 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:11.172416 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:11.172416 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:11.172416 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:11.172416 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:11.172416 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:11.173604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:11.173604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:11.173604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:11.173604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:11.173604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:11.173604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:11.173604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:11.173604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:11.173604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:11.173604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:11.173604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 27 01:12:11.173604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:11.173604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 27 01:12:11.173604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:11.173604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 305/607] 10.222.0.42 () {56 vars in 1132 bytes} [Thu Oct 27 01:12:11 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:11.174781 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4c0af5-1857-40f3-821f-4b08710d92d2 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:11.213564 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 303/608] 10.222.0.42 () {58 vars in 1282 bytes} [Thu Oct 27 01:12:10 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 459 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 27 01:12:11.214707 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e35da9-bcf7-441b-966c-e1ab5318c199 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:11.293482 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4c0af5-1857-40f3-821f-4b08710d92d2 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:11.330529 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c50ef7-a3f6-46f8-9459-5265f983a5e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:11.331753 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c50ef7-a3f6-46f8-9459-5265f983a5e9 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:11.331900 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e35da9-bcf7-441b-966c-e1ab5318c199 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:11.332217 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c50ef7-a3f6-46f8-9459-5265f983a5e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:11.332584 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c50ef7-a3f6-46f8-9459-5265f983a5e9 None admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:11.333461 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79c50ef7-a3f6-46f8-9459-5265f983a5e9 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:11.333865 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79c50ef7-a3f6-46f8-9459-5265f983a5e9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:11.335396 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79c50ef7-a3f6-46f8-9459-5265f983a5e9 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:11.344781 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:11.355385 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 306/609] 10.222.0.42 () {56 vars in 1149 bytes} [Thu Oct 27 01:12:11 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:11.356415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c50ef7-a3f6-46f8-9459-5265f983a5e9 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:11.371048 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f4d54a8-021d-4145-bdb8-dc7a0519b48a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:11.371798 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4d54a8-021d-4145-bdb8-dc7a0519b48a None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:11.372188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4d54a8-021d-4145-bdb8-dc7a0519b48a None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:11.372588 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4d54a8-021d-4145-bdb8-dc7a0519b48a None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:11.373365 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f4d54a8-021d-4145-bdb8-dc7a0519b48a None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:11.373778 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f4d54a8-021d-4145-bdb8-dc7a0519b48a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:11.375211 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f4d54a8-021d-4145-bdb8-dc7a0519b48a None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:11.379876 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 304/610] 10.222.0.42 () {56 vars in 1126 bytes} [Thu Oct 27 01:12:11 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:11.381045 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf22da8-755f-4d88-b5a1-ed0c23fb59ff None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:11.381352 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf22da8-755f-4d88-b5a1-ed0c23fb59ff None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:11.381626 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf22da8-755f-4d88-b5a1-ed0c23fb59ff None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:11.382967 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 305/611] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:11 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:11.383747 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf22da8-755f-4d88-b5a1-ed0c23fb59ff None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:11.401746 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:11.464475 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c50ef7-a3f6-46f8-9459-5265f983a5e9 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:11.482156 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf22da8-755f-4d88-b5a1-ed0c23fb59ff None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:11.498594 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17d801a-414f-4082-a62d-bbce0c1fe807 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:11.499214 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17d801a-414f-4082-a62d-bbce0c1fe807 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:11.499487 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17d801a-414f-4082-a62d-bbce0c1fe807 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:11.499761 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17d801a-414f-4082-a62d-bbce0c1fe807 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:11.509556 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17d801a-414f-4082-a62d-bbce0c1fe807 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:11.510999 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17d801a-414f-4082-a62d-bbce0c1fe807 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:11.515961 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2a95e4-76e7-4c7f-9acd-32c713d2abad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:11.516555 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2a95e4-76e7-4c7f-9acd-32c713d2abad None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:11.516830 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2a95e4-76e7-4c7f-9acd-32c713d2abad None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:11.517125 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2a95e4-76e7-4c7f-9acd-32c713d2abad None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:11.525868 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-d17d801a-414f-4082-a62d-bbce0c1fe807 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:12:11.525868 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:11.525868 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:11.525868 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:11.525868 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:11.525868 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:11.525868 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:11.525868 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:11.525868 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:11.525868 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:11.525868 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:11.525868 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:11.525868 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:12:11.525868 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:12:11.525868 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:12:11.525868 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:12:11.526952 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:11.526952 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:11.526952 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:11.526952 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:11.526952 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:11.526952 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:11.526952 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:11.526952 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:11.526952 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:11.526952 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:11.526952 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:11.526952 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:11.526952 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:11.526952 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:11.526952 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:11.526952 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:11.526952 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:12:11.527857 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:12:11.527857 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:12:11.527857 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:11.527857 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 307/612] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:12:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:11.528154 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d12d23-615c-4b6f-88b3-6dbc89364338 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:11.528422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d12d23-615c-4b6f-88b3-6dbc89364338 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:11.528697 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d12d23-615c-4b6f-88b3-6dbc89364338 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:11.529920 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 308/613] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:11 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:11.530594 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d12d23-615c-4b6f-88b3-6dbc89364338 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:11.583701 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:11.590635 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2a95e4-76e7-4c7f-9acd-32c713d2abad None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:11.592078 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2a95e4-76e7-4c7f-9acd-32c713d2abad None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:11.609318 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:11.633696 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d12d23-615c-4b6f-88b3-6dbc89364338 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:11.666525 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df333d52-706c-46be-88a9-c2d674676bf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:11.667126 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df333d52-706c-46be-88a9-c2d674676bf7 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:11.667408 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df333d52-706c-46be-88a9-c2d674676bf7 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:11.667680 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df333d52-706c-46be-88a9-c2d674676bf7 None admin] PATH_INFO: `/v3/services` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:11.668308 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df333d52-706c-46be-88a9-c2d674676bf7 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:11.668607 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df333d52-706c-46be-88a9-c2d674676bf7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:11.669991 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df333d52-706c-46be-88a9-c2d674676bf7 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:11.674439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 309/614] 10.222.0.42 () {56 vars in 1126 bytes} [Thu Oct 27 01:12:11 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:11.676330 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba214b2-8d68-4f64-a5c4-6a717999b876 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:11.676615 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba214b2-8d68-4f64-a5c4-6a717999b876 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:11.676895 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba214b2-8d68-4f64-a5c4-6a717999b876 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:11.696643 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-aba214b2-8d68-4f64-a5c4-6a717999b876 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:11.697938 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-aba214b2-8d68-4f64-a5c4-6a717999b876 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:11.705676 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 306/615] 10.222.0.42 () {58 vars in 1282 bytes} [Thu Oct 27 01:12:11 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 27 01:12:11.705676 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2a95e4-76e7-4c7f-9acd-32c713d2abad None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:11.781808 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 310/616] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:11 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:12:11.783489 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96afa8b1-c1bb-4063-aaf9-075b68411bd1 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:11.783775 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96afa8b1-c1bb-4063-aaf9-075b68411bd1 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:11.784103 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96afa8b1-c1bb-4063-aaf9-075b68411bd1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:11.807799 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-96afa8b1-c1bb-4063-aaf9-075b68411bd1 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:11.809413 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-96afa8b1-c1bb-4063-aaf9-075b68411bd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:11.839110 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2a95e4-76e7-4c7f-9acd-32c713d2abad None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:11.871028 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:11.873222 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:11.887773 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22eaab5-6a99-4e75-96eb-35a50eb86eea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:11.888390 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22eaab5-6a99-4e75-96eb-35a50eb86eea None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:11.888669 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22eaab5-6a99-4e75-96eb-35a50eb86eea None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:11.889366 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22eaab5-6a99-4e75-96eb-35a50eb86eea None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:11.889737 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e22eaab5-6a99-4e75-96eb-35a50eb86eea None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:11.890064 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e22eaab5-6a99-4e75-96eb-35a50eb86eea None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:11.892330 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e22eaab5-6a99-4e75-96eb-35a50eb86eea None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:11.905816 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 311/617] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:11 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:12:11.906826 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96afa8b1-c1bb-4063-aaf9-075b68411bd1 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:11.909145 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 307/618] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:12:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:11.914747 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22eaab5-6a99-4e75-96eb-35a50eb86eea None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:12.021248 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96afa8b1-c1bb-4063-aaf9-075b68411bd1 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:12.061084 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b96b18a-9dae-4747-af8c-23d638e9ee55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:12.061084 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b96b18a-9dae-4747-af8c-23d638e9ee55 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:12.061084 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b96b18a-9dae-4747-af8c-23d638e9ee55 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:12.061084 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b96b18a-9dae-4747-af8c-23d638e9ee55 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:12.064369 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22eaab5-6a99-4e75-96eb-35a50eb86eea None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:12.073992 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b96b18a-9dae-4747-af8c-23d638e9ee55 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:12.075559 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b96b18a-9dae-4747-af8c-23d638e9ee55 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:12.108538 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-5b96b18a-9dae-4747-af8c-23d638e9ee55 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:12:12.108538 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:12.108538 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:12.108538 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:12.108538 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:12.108538 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:12.108538 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:12.108538 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:12.108538 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:12.108538 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:12.108538 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:12.108538 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:12.108538 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:12:12.108538 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:12:12.108538 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:12:12.108538 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:12:12.109580 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:12.109580 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:12.109580 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:12.109580 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:12.109580 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:12.109580 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:12.109580 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:12.109580 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:12.109580 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:12.109580 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:12.109580 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:12.109580 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:12.109580 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:12.109580 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:12.109580 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:12.109580 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:12.109580 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:12:12.110479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:12:12.110479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:12:12.110479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:12.112692 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 312/619] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:12:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:12.117603 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b96b18a-9dae-4747-af8c-23d638e9ee55 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:12.136696 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32cba548-a5f1-4dea-9976-6bb5acab0cd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:12.137408 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cba548-a5f1-4dea-9976-6bb5acab0cd0 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:12.137686 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cba548-a5f1-4dea-9976-6bb5acab0cd0 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:12.137951 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cba548-a5f1-4dea-9976-6bb5acab0cd0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:12.152799 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32cba548-a5f1-4dea-9976-6bb5acab0cd0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:12.158068 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32cba548-a5f1-4dea-9976-6bb5acab0cd0 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:12.174178 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:12.176468 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-32cba548-a5f1-4dea-9976-6bb5acab0cd0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:12:12.176468 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:12.176468 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:12.176468 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:12.176468 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:12.176468 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:12.176468 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:12.176468 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:12.176468 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:12.176468 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:12.176468 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:12.176468 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:12.176468 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 27 01:12:12.176468 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 27 01:12:12.176468 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 27 01:12:12.178185 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 27 01:12:12.178185 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:12.178185 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:12.178185 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:12.178185 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:12.178185 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:12.178185 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:12.178185 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:12.178185 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:12.178185 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:12.178185 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:12.178185 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:12.178185 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:12.178185 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:12.178185 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:12.178185 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:12.178185 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:12.179464 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 27 01:12:12.179464 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 27 01:12:12.179464 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 27 01:12:12.179464 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 27 01:12:12.179464 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 27 01:12:12.179464 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 27 01:12:12.179464 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:12:12.179464 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:12.179464 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 308/620] 10.222.0.42 () {56 vars in 1111 bytes} [Thu Oct 27 01:12:11 2022] GET /identity/v3/projects/service => generated 88 bytes in 264 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:12.184360 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32cba548-a5f1-4dea-9976-6bb5acab0cd0 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:12.226767 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:12.263659 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b96b18a-9dae-4747-af8c-23d638e9ee55 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:12.290156 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32cba548-a5f1-4dea-9976-6bb5acab0cd0 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:12.297404 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a23c527-7865-4976-aa52-74a797cb83c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:12.298011 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a23c527-7865-4976-aa52-74a797cb83c9 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:12.298284 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a23c527-7865-4976-aa52-74a797cb83c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:12.298558 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a23c527-7865-4976-aa52-74a797cb83c9 None admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:12.299226 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a23c527-7865-4976-aa52-74a797cb83c9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:12.299525 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a23c527-7865-4976-aa52-74a797cb83c9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:12.301488 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a23c527-7865-4976-aa52-74a797cb83c9 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:12.314115 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 313/621] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:12:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:12.314817 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a23c527-7865-4976-aa52-74a797cb83c9 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:12.334765 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b28067-e210-487e-95a7-f6ba60cb913c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:12.335702 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b28067-e210-487e-95a7-f6ba60cb913c None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:12.336199 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b28067-e210-487e-95a7-f6ba60cb913c None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:12.337120 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b28067-e210-487e-95a7-f6ba60cb913c None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:12.338184 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b28067-e210-487e-95a7-f6ba60cb913c None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:12.339746 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b28067-e210-487e-95a7-f6ba60cb913c None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:12.349655 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-b3b28067-e210-487e-95a7-f6ba60cb913c None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 27 01:12:12.349655 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:12.349655 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 27 01:12:12.349655 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 27 01:12:12.349655 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 27 01:12:12.349655 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 27 01:12:12.349655 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 27 01:12:12.349655 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:12.349655 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 27 01:12:12.349655 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:12.349655 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 27 01:12:12.349655 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:12.349655 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:12.349655 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:12.349655 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:12.349655 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:12.349655 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:12.350663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:12.350663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:12.350663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:12.350663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:12.350663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:12.350663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:12.350663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 27 01:12:12.350663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 27 01:12:12.350663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 27 01:12:12.350663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 27 01:12:12.350663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:12.350663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:12.350663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:12.350663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:12.350663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:12.350663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:12.351536 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:12.351536 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:12.351536 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:12.351536 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:12.351536 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:12.351536 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:12.351536 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:12.351536 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:12.351536 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:12.351536 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:12.351536 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 27 01:12:12.351536 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:12.351536 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 27 01:12:12.351536 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:12.351536 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 309/622] 10.222.0.42 () {56 vars in 1129 bytes} [Thu Oct 27 01:12:12 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:12.352352 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b28067-e210-487e-95a7-f6ba60cb913c None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:12.411273 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:12.428317 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a23c527-7865-4976-aa52-74a797cb83c9 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:12.455789 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b28067-e210-487e-95a7-f6ba60cb913c None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:12.458966 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:12.465349 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8bae175-ab2f-4efa-b8a3-38d68e7f3ec4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:12.466038 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8bae175-ab2f-4efa-b8a3-38d68e7f3ec4 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:12.466312 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8bae175-ab2f-4efa-b8a3-38d68e7f3ec4 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:12.466579 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8bae175-ab2f-4efa-b8a3-38d68e7f3ec4 None admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:12.467265 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8bae175-ab2f-4efa-b8a3-38d68e7f3ec4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:12.467564 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8bae175-ab2f-4efa-b8a3-38d68e7f3ec4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:12.468974 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8bae175-ab2f-4efa-b8a3-38d68e7f3ec4 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:12.480723 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 314/623] 10.222.0.42 () {56 vars in 1148 bytes} [Thu Oct 27 01:12:12 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:12.481556 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8bae175-ab2f-4efa-b8a3-38d68e7f3ec4 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:12.501728 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba065a68-2afe-464f-857b-6a9044007664 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:12.502605 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba065a68-2afe-464f-857b-6a9044007664 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:12.502887 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba065a68-2afe-464f-857b-6a9044007664 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:12.503701 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba065a68-2afe-464f-857b-6a9044007664 None admin] PATH_INFO: `/v3/services/network` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:12.504287 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba065a68-2afe-464f-857b-6a9044007664 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:12.505687 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba065a68-2afe-464f-857b-6a9044007664 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:12.515772 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-ba065a68-2afe-464f-857b-6a9044007664 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 27 01:12:12.515772 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:12.515772 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 27 01:12:12.515772 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 27 01:12:12.515772 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 27 01:12:12.515772 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 27 01:12:12.515772 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 27 01:12:12.515772 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:12.515772 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 27 01:12:12.515772 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:12.515772 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 27 01:12:12.515772 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:12.515772 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:12.515772 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:12.515772 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:12.515772 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:12.515772 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:12.516769 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:12.516769 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:12.516769 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:12.516769 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:12.516769 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:12.516769 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:12.516769 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 27 01:12:12.516769 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 27 01:12:12.516769 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 27 01:12:12.516769 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 27 01:12:12.516769 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:12.516769 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:12.516769 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:12.516769 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:12.516769 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:12.516769 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:12.517634 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:12.517634 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:12.517634 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:12.517634 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:12.517634 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:12.517634 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:12.517634 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:12.517634 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:12.517634 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:12.517634 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:12.517634 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 27 01:12:12.517634 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:12.517634 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 27 01:12:12.517634 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:12.522113 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 310/624] 10.222.0.42 () {56 vars in 1111 bytes} [Thu Oct 27 01:12:12 2022] GET /identity/v3/services/network => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:12.522993 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba065a68-2afe-464f-857b-6a9044007664 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:12.589971 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8bae175-ab2f-4efa-b8a3-38d68e7f3ec4 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:12.628907 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73cc8ba-518c-4657-9537-9c25171f5241 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:12.629604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73cc8ba-518c-4657-9537-9c25171f5241 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:12.629721 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba065a68-2afe-464f-857b-6a9044007664 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:12.629907 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73cc8ba-518c-4657-9537-9c25171f5241 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:12.630141 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73cc8ba-518c-4657-9537-9c25171f5241 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:12.636897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73cc8ba-518c-4657-9537-9c25171f5241 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:12.638339 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73cc8ba-518c-4657-9537-9c25171f5241 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:12.650578 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-f73cc8ba-518c-4657-9537-9c25171f5241 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:12:12.650578 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:12.650578 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:12.650578 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:12.650578 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:12.650578 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:12.650578 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:12.650578 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:12.650578 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:12.650578 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:12.650578 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:12.650578 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:12.650578 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 27 01:12:12.650578 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 27 01:12:12.650578 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 27 01:12:12.651468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 27 01:12:12.651468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:12.651468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:12.651468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:12.651468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:12.651468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:12.651468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:12.651468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:12.651468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:12.651468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:12.651468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:12.651468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:12.651468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:12.651468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:12.651468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:12.651468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:12.651468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:12.652372 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 27 01:12:12.652372 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 27 01:12:12.652372 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 27 01:12:12.652372 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 27 01:12:12.652372 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 27 01:12:12.652372 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 27 01:12:12.652372 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:12:12.652372 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:12.652372 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 315/625] 10.222.0.42 () {56 vars in 1111 bytes} [Thu Oct 27 01:12:12 2022] GET /identity/v3/projects/service => generated 88 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:12.652372 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73cc8ba-518c-4657-9537-9c25171f5241 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:12.666941 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:12.671196 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa4712a-74f8-49ba-b966-accc91c8b837 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:12.671805 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa4712a-74f8-49ba-b966-accc91c8b837 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:12.672073 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa4712a-74f8-49ba-b966-accc91c8b837 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:12.672342 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa4712a-74f8-49ba-b966-accc91c8b837 None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:12.673049 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa4712a-74f8-49ba-b966-accc91c8b837 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:12.673307 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa4712a-74f8-49ba-b966-accc91c8b837 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:12.674652 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa4712a-74f8-49ba-b966-accc91c8b837 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:12.684561 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 311/626] 10.222.0.42 () {56 vars in 1124 bytes} [Thu Oct 27 01:12:12 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:12.685418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa4712a-74f8-49ba-b966-accc91c8b837 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:12.718820 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:12.765406 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73cc8ba-518c-4657-9537-9c25171f5241 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:12.796738 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa4712a-74f8-49ba-b966-accc91c8b837 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:12.801915 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed5d4c69-d3ee-41cd-88b4-87aa58c33df8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:12.802519 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5d4c69-d3ee-41cd-88b4-87aa58c33df8 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:12.802797 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5d4c69-d3ee-41cd-88b4-87aa58c33df8 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:12.803063 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5d4c69-d3ee-41cd-88b4-87aa58c33df8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:12.803780 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed5d4c69-d3ee-41cd-88b4-87aa58c33df8 None admin] RBAC: Adding query filter params (role.id=414286927fdd44688107ff5e10235acf, scope.project.id=23aad848d47c4d27ad4c138ff869e40f, user.id=77f2cf9e0c3541f9bccfb7e213d88609) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:12.804080 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed5d4c69-d3ee-41cd-88b4-87aa58c33df8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:12.806004 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed5d4c69-d3ee-41cd-88b4-87aa58c33df8 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:12.816989 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 316/627] 10.222.0.42 () {56 vars in 1374 bytes} [Thu Oct 27 01:12:12 2022] GET /identity/v3/role_assignments?user.id=77f2cf9e0c3541f9bccfb7e213d88609&scope.project.id=23aad848d47c4d27ad4c138ff869e40f&role.id=414286927fdd44688107ff5e10235acf => generated 608 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:12.817680 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed5d4c69-d3ee-41cd-88b4-87aa58c33df8 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:12.837651 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e97a188-e719-49ca-9c25-3b13520cb5ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:12.838274 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e97a188-e719-49ca-9c25-3b13520cb5ab None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:12.838600 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e97a188-e719-49ca-9c25-3b13520cb5ab None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:12.838875 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e97a188-e719-49ca-9c25-3b13520cb5ab None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:12.839576 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e97a188-e719-49ca-9c25-3b13520cb5ab None admin] RBAC: Adding query filter params (name=network) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:12.839886 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e97a188-e719-49ca-9c25-3b13520cb5ab None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:12.841365 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e97a188-e719-49ca-9c25-3b13520cb5ab None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:12.852149 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 312/628] 10.222.0.42 () {56 vars in 1112 bytes} [Thu Oct 27 01:12:12 2022] GET /identity/v3/services?name=network => generated 124 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:12.853172 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e97a188-e719-49ca-9c25-3b13520cb5ab None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:12.916373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:12.965270 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed5d4c69-d3ee-41cd-88b4-87aa58c33df8 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:12.985663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e97a188-e719-49ca-9c25-3b13520cb5ab None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:12.989490 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:13.007940 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537a728a-5de1-4e3a-a87a-b8ef98abd655 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:13.008851 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537a728a-5de1-4e3a-a87a-b8ef98abd655 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:13.009375 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537a728a-5de1-4e3a-a87a-b8ef98abd655 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:13.009850 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537a728a-5de1-4e3a-a87a-b8ef98abd655 None admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:13.011500 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-537a728a-5de1-4e3a-a87a-b8ef98abd655 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:13.012023 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-537a728a-5de1-4e3a-a87a-b8ef98abd655 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:13.013689 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-537a728a-5de1-4e3a-a87a-b8ef98abd655 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:13.026873 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 317/629] 10.222.0.42 () {56 vars in 1148 bytes} [Thu Oct 27 01:12:12 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:13.028008 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537a728a-5de1-4e3a-a87a-b8ef98abd655 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:13.031796 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a94204-6ec4-43c6-8cd8-72abeb61a946 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:13.032650 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a94204-6ec4-43c6-8cd8-72abeb61a946 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:13.033162 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a94204-6ec4-43c6-8cd8-72abeb61a946 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:13.033668 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a94204-6ec4-43c6-8cd8-72abeb61a946 None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:13.034975 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a94204-6ec4-43c6-8cd8-72abeb61a946 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:13.035285 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a94204-6ec4-43c6-8cd8-72abeb61a946 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:13.036666 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a94204-6ec4-43c6-8cd8-72abeb61a946 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:13.043288 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 313/630] 10.222.0.42 () {56 vars in 1124 bytes} [Thu Oct 27 01:12:12 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:13.043946 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a94204-6ec4-43c6-8cd8-72abeb61a946 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:13.144804 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537a728a-5de1-4e3a-a87a-b8ef98abd655 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:13.163654 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a94204-6ec4-43c6-8cd8-72abeb61a946 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:13.177390 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:13.182998 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727f1b12-9556-4bda-a5c2-774f44627aab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:13.183773 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727f1b12-9556-4bda-a5c2-774f44627aab None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:13.184391 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727f1b12-9556-4bda-a5c2-774f44627aab None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:13.184735 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727f1b12-9556-4bda-a5c2-774f44627aab None admin] PATH_INFO: `/v3/services` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:13.185503 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727f1b12-9556-4bda-a5c2-774f44627aab None admin] RBAC: Adding query filter params (type=network) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:13.185853 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727f1b12-9556-4bda-a5c2-774f44627aab None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:13.187388 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727f1b12-9556-4bda-a5c2-774f44627aab None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:13.193298 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 318/631] 10.222.0.42 () {56 vars in 1112 bytes} [Thu Oct 27 01:12:13 2022] GET /identity/v3/services?type=network => generated 350 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:13.194222 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727f1b12-9556-4bda-a5c2-774f44627aab None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:13.210339 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea6077a-2b5a-495c-b4a0-ff31a9479e10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:13.211417 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea6077a-2b5a-495c-b4a0-ff31a9479e10 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:13.211697 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea6077a-2b5a-495c-b4a0-ff31a9479e10 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:13.211971 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea6077a-2b5a-495c-b4a0-ff31a9479e10 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:13.213188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ea6077a-2b5a-495c-b4a0-ff31a9479e10 None admin] RBAC: Adding query filter params (role.id=1b8f8dd9e8584053b048032db63254a4, scope.project.id=23aad848d47c4d27ad4c138ff869e40f, user.id=7a3f8b5eb09f4ba29f4bd343bd91f6a2) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:13.213634 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ea6077a-2b5a-495c-b4a0-ff31a9479e10 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:13.215050 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ea6077a-2b5a-495c-b4a0-ff31a9479e10 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:13.222448 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 314/632] 10.222.0.42 () {56 vars in 1374 bytes} [Thu Oct 27 01:12:13 2022] GET /identity/v3/role_assignments?user.id=7a3f8b5eb09f4ba29f4bd343bd91f6a2&scope.project.id=23aad848d47c4d27ad4c138ff869e40f&role.id=1b8f8dd9e8584053b048032db63254a4 => generated 608 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:13.223158 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea6077a-2b5a-495c-b4a0-ff31a9479e10 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:13.253820 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:13.375475 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727f1b12-9556-4bda-a5c2-774f44627aab None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:13.414869 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea6077a-2b5a-495c-b4a0-ff31a9479e10 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:13.441656 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678996c4-9e3d-4ddb-a609-bd33915b2f64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:13.442629 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678996c4-9e3d-4ddb-a609-bd33915b2f64 None admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:13.443147 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678996c4-9e3d-4ddb-a609-bd33915b2f64 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:13.443683 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678996c4-9e3d-4ddb-a609-bd33915b2f64 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:13.444544 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-678996c4-9e3d-4ddb-a609-bd33915b2f64 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:13.446218 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-678996c4-9e3d-4ddb-a609-bd33915b2f64 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:13.464740 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de7fdcc-9bd2-4d1d-a9ca-f0ba221353e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:13.465397 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de7fdcc-9bd2-4d1d-a9ca-f0ba221353e0 None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:13.465684 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de7fdcc-9bd2-4d1d-a9ca-f0ba221353e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:13.467064 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de7fdcc-9bd2-4d1d-a9ca-f0ba221353e0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:13.467064 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de7fdcc-9bd2-4d1d-a9ca-f0ba221353e0 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:13.467979 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de7fdcc-9bd2-4d1d-a9ca-f0ba221353e0 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:13.490728 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 319/633] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:12:13 2022] POST /identity/v3/endpoints => generated 346 bytes in 297 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:12:13.497467 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 315/634] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:12:13 2022] POST /identity/v3/endpoints => generated 337 bytes in 274 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:12:13.972574 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a66727-79a1-4a75-885a-f1b003078aa5 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:13.973594 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a66727-79a1-4a75-885a-f1b003078aa5 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:13.973594 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a66727-79a1-4a75-885a-f1b003078aa5 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:13.975041 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 320/635] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:13.980537 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2712fb-9394-4c4e-aead-a8e3cb8c2588 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:13.980943 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2712fb-9394-4c4e-aead-a8e3cb8c2588 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:13.981161 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2712fb-9394-4c4e-aead-a8e3cb8c2588 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:13.999749 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-aa2712fb-9394-4c4e-aead-a8e3cb8c2588 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:14.000934 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-aa2712fb-9394-4c4e-aead-a8e3cb8c2588 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:14.087544 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 316/636] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:13 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:12:14.362529 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a66727-79a1-4a75-885a-f1b003078aa5 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:14.419636 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:14.479281 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a66727-79a1-4a75-885a-f1b003078aa5 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:14.512818 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b6d5f8-4333-4b3b-9490-6f2d00a81d1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:14.513443 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b6d5f8-4333-4b3b-9490-6f2d00a81d1b None admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:14.513721 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b6d5f8-4333-4b3b-9490-6f2d00a81d1b None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:14.513989 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b6d5f8-4333-4b3b-9490-6f2d00a81d1b None admin] PATH_INFO: `/v3/services` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:14.514648 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b6d5f8-4333-4b3b-9490-6f2d00a81d1b None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:14.516001 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b6d5f8-4333-4b3b-9490-6f2d00a81d1b None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:14.532943 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 321/637] 10.222.0.42 () {60 vars in 1141 bytes} [Thu Oct 27 01:12:14 2022] POST /identity/v3/services => generated 269 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:12:15.245429 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb9684a-2e55-4306-87f4-2fc1965cc240 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:15.245429 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb9684a-2e55-4306-87f4-2fc1965cc240 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:15.245429 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb9684a-2e55-4306-87f4-2fc1965cc240 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:15.246575 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 317/638] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:15.252112 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485f85ea-d331-449a-87b3-bf76ee0f05ad None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:15.253476 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485f85ea-d331-449a-87b3-bf76ee0f05ad None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:15.254536 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485f85ea-d331-449a-87b3-bf76ee0f05ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:15.273855 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-485f85ea-d331-449a-87b3-bf76ee0f05ad None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:15.275368 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-485f85ea-d331-449a-87b3-bf76ee0f05ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:15.357161 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 322/639] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:15 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:12:15.632161 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb9684a-2e55-4306-87f4-2fc1965cc240 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:15.637629 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:15.727527 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2f1ee7-85d9-426a-9932-80c97bdaaa82 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:15.727846 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2f1ee7-85d9-426a-9932-80c97bdaaa82 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:15.728085 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2f1ee7-85d9-426a-9932-80c97bdaaa82 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:15.729696 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 323/640] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:15.734653 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e93c204-7952-4fbd-93f7-1373f56cca5d None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:15.735186 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e93c204-7952-4fbd-93f7-1373f56cca5d None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:15.735637 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e93c204-7952-4fbd-93f7-1373f56cca5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:15.748293 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb9684a-2e55-4306-87f4-2fc1965cc240 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:15.755942 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-2e93c204-7952-4fbd-93f7-1373f56cca5d None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:15.757582 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-2e93c204-7952-4fbd-93f7-1373f56cca5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:15.792994 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ecd83dd-9719-44a5-a50e-e5b3c2aceca8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:15.793455 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:15.793638 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ecd83dd-9719-44a5-a50e-e5b3c2aceca8 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:15.793917 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ecd83dd-9719-44a5-a50e-e5b3c2aceca8 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:15.794181 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ecd83dd-9719-44a5-a50e-e5b3c2aceca8 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:15.794787 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ecd83dd-9719-44a5-a50e-e5b3c2aceca8 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:15.796300 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ecd83dd-9719-44a5-a50e-e5b3c2aceca8 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:15.812816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-7ecd83dd-9719-44a5-a50e-e5b3c2aceca8 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 27 01:12:15.812816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:15.812816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 27 01:12:15.812816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 27 01:12:15.812816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 27 01:12:15.812816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 27 01:12:15.812816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 27 01:12:15.812816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:15.812816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 27 01:12:15.812816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:15.812816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 27 01:12:15.812816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:15.812816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:15.812816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:15.812816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:15.812816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:15.812816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:15.813919 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:15.813919 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:15.813919 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:15.813919 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:15.813919 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:15.813919 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:15.813919 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 27 01:12:15.813919 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 27 01:12:15.813919 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 27 01:12:15.813919 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 27 01:12:15.813919 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:15.813919 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:15.813919 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:15.813919 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:15.813919 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:15.813919 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:15.814760 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:15.814760 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:15.814760 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:15.814760 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:15.814760 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:15.814760 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:15.814760 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:15.814760 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:15.814760 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:15.814760 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:15.814760 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 27 01:12:15.814760 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:15.814760 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 27 01:12:15.814760 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:15.814760 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 318/641] 10.222.0.42 () {56 vars in 1126 bytes} [Thu Oct 27 01:12:15 2022] GET /identity/v3/services/object-store => generated 93 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:15.820727 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ecd83dd-9719-44a5-a50e-e5b3c2aceca8 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:15.849165 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 324/642] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:15 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 27 01:12:15.875559 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb6cf3e-e40e-4dfe-8a5f-7242fa449e2d None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:15.875881 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb6cf3e-e40e-4dfe-8a5f-7242fa449e2d None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:15.876111 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb6cf3e-e40e-4dfe-8a5f-7242fa449e2d None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:15.877743 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 325/643] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 27 01:12:15.882949 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e048337-f1a4-4a69-b32e-822875e6e225 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:15.883162 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e048337-f1a4-4a69-b32e-822875e6e225 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:15.883491 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e048337-f1a4-4a69-b32e-822875e6e225 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:15.897459 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:15.901937 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-1e048337-f1a4-4a69-b32e-822875e6e225 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:15.903252 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-1e048337-f1a4-4a69-b32e-822875e6e225 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:15.936197 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ecd83dd-9719-44a5-a50e-e5b3c2aceca8 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:15.981022 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 326/644] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:15 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 27 01:12:15.990261 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7367086-1f53-49b3-a950-8870defa4a8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:15.990917 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7367086-1f53-49b3-a950-8870defa4a8d None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:15.991194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7367086-1f53-49b3-a950-8870defa4a8d None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:15.991459 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7367086-1f53-49b3-a950-8870defa4a8d None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:15.992084 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7367086-1f53-49b3-a950-8870defa4a8d None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:15.992379 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7367086-1f53-49b3-a950-8870defa4a8d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:15.993767 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7367086-1f53-49b3-a950-8870defa4a8d None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:16.003801 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 319/645] 10.222.0.42 () {56 vars in 1122 bytes} [Thu Oct 27 01:12:15 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:12:16.005251 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e048337-f1a4-4a69-b32e-822875e6e225 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:16.104997 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:16.111231 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e048337-f1a4-4a69-b32e-822875e6e225 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:16.122522 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7367086-1f53-49b3-a950-8870defa4a8d None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:16.155825 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaaf4635-ffdd-48cd-8de4-f020388ca70e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:16.156726 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaaf4635-ffdd-48cd-8de4-f020388ca70e None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:16.157212 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaaf4635-ffdd-48cd-8de4-f020388ca70e None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:16.157687 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaaf4635-ffdd-48cd-8de4-f020388ca70e None admin] PATH_INFO: `/v3/services` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:16.159630 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaaf4635-ffdd-48cd-8de4-f020388ca70e None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:16.160145 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaaf4635-ffdd-48cd-8de4-f020388ca70e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:16.161726 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaaf4635-ffdd-48cd-8de4-f020388ca70e None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:16.168321 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 327/646] 10.222.0.42 () {56 vars in 1122 bytes} [Thu Oct 27 01:12:16 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:16.239565 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7367086-1f53-49b3-a950-8870defa4a8d None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:16.274105 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:16.281248 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaaf4635-ffdd-48cd-8de4-f020388ca70e None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:16.283368 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faed4cdd-c25e-45af-ab14-4933a1ce1f3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:16.284718 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faed4cdd-c25e-45af-ab14-4933a1ce1f3d None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:16.286447 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faed4cdd-c25e-45af-ab14-4933a1ce1f3d None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:16.286983 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faed4cdd-c25e-45af-ab14-4933a1ce1f3d None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:16.302357 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faed4cdd-c25e-45af-ab14-4933a1ce1f3d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:12:16.309403 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-faed4cdd-c25e-45af-ab14-4933a1ce1f3d None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 27 01:12:16.309403 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:16.309403 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:16.309403 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:16.309403 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:16.309403 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:16.309403 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:16.309403 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:16.309403 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:16.309403 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:16.309403 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:16.309403 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:16.309403 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:12:16.309403 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:12:16.309403 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:12:16.309403 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:12:16.310418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:12:16.310418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:12:16.310418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:12:16.310418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:12:16.310418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:16.310418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:16.310418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:16.310418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:16.310418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:16.310418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:16.310418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:16.310418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:16.310418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:16.310418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:16.310418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:16.310418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:16.311266 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:16.311266 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:16.311266 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:16.311266 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:16.311266 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:12:16.311266 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:12:16.311266 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:12:16.311266 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:12:16.311266 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:12:16.311266 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:12:16.311266 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 27 01:12:16.311266 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:16.314437 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 320/647] 10.222.0.42 () {56 vars in 1120 bytes} [Thu Oct 27 01:12:16 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 27 01:12:16.320427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faed4cdd-c25e-45af-ab14-4933a1ce1f3d None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:16.405566 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaaf4635-ffdd-48cd-8de4-f020388ca70e None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:16.446012 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faed4cdd-c25e-45af-ab14-4933a1ce1f3d None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:16.449659 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1411c0d4-3716-446d-8746-54f0d6ee417b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:16.450290 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1411c0d4-3716-446d-8746-54f0d6ee417b None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:16.450562 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1411c0d4-3716-446d-8746-54f0d6ee417b None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:16.450828 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1411c0d4-3716-446d-8746-54f0d6ee417b None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:16.451409 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1411c0d4-3716-446d-8746-54f0d6ee417b None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:16.452778 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1411c0d4-3716-446d-8746-54f0d6ee417b None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:16.462247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-1411c0d4-3716-446d-8746-54f0d6ee417b None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 27 01:12:16.462247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:16.462247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 27 01:12:16.462247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 27 01:12:16.462247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 27 01:12:16.462247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 27 01:12:16.462247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 27 01:12:16.462247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:16.462247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 27 01:12:16.462247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:16.462247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 27 01:12:16.462247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:16.462247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:16.462247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:16.462247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:16.462247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:16.462247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:16.463247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:16.463247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:16.463247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:16.463247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:16.463247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:16.463247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:16.463247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 27 01:12:16.463247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 27 01:12:16.463247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 27 01:12:16.463247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 27 01:12:16.463247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:16.463247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:16.463247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:16.463247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:16.463247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:16.463247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:16.464446 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:16.464446 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:16.464446 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:16.464446 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:16.464446 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:16.464446 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:16.464446 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:16.464446 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:16.464446 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:16.464446 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:16.464446 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 27 01:12:16.464446 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:16.464446 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 27 01:12:16.464446 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:16.464446 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 328/648] 10.222.0.42 () {56 vars in 1114 bytes} [Thu Oct 27 01:12:16 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:16.469225 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1411c0d4-3716-446d-8746-54f0d6ee417b None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:16.492765 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f73951c-aa2e-4959-a353-accf14e0cc91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:16.493690 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f73951c-aa2e-4959-a353-accf14e0cc91 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:16.494107 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f73951c-aa2e-4959-a353-accf14e0cc91 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:16.494598 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f73951c-aa2e-4959-a353-accf14e0cc91 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:16.495462 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f73951c-aa2e-4959-a353-accf14e0cc91 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:16.496030 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f73951c-aa2e-4959-a353-accf14e0cc91 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:16.497609 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f73951c-aa2e-4959-a353-accf14e0cc91 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:16.501256 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:16.509582 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 321/649] 10.222.0.42 () {56 vars in 1115 bytes} [Thu Oct 27 01:12:16 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:12:16.517105 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f73951c-aa2e-4959-a353-accf14e0cc91 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:16.590345 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1411c0d4-3716-446d-8746-54f0d6ee417b None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:16.590557 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:16.630010 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d0388b-48d5-480f-bd0c-38bec97ca185 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:16.630966 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d0388b-48d5-480f-bd0c-38bec97ca185 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:16.631389 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d0388b-48d5-480f-bd0c-38bec97ca185 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:16.631832 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d0388b-48d5-480f-bd0c-38bec97ca185 None admin] PATH_INFO: `/v3/services` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:16.632603 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d0388b-48d5-480f-bd0c-38bec97ca185 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:16.633048 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d0388b-48d5-480f-bd0c-38bec97ca185 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:16.634669 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d0388b-48d5-480f-bd0c-38bec97ca185 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:16.645473 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 329/650] 10.222.0.42 () {56 vars in 1114 bytes} [Thu Oct 27 01:12:16 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:16.646576 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d0388b-48d5-480f-bd0c-38bec97ca185 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:16.646965 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f73951c-aa2e-4959-a353-accf14e0cc91 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:16.695169 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67da44c2-1838-4291-87a1-7bf5bc729502 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:16.695928 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67da44c2-1838-4291-87a1-7bf5bc729502 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:16.696387 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67da44c2-1838-4291-87a1-7bf5bc729502 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:16.696704 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67da44c2-1838-4291-87a1-7bf5bc729502 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:16.716370 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67da44c2-1838-4291-87a1-7bf5bc729502 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:16.718246 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67da44c2-1838-4291-87a1-7bf5bc729502 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:16.739900 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-67da44c2-1838-4291-87a1-7bf5bc729502 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:12:16.739900 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:16.739900 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:16.739900 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:16.739900 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:16.739900 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:16.739900 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:16.739900 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:16.739900 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:16.739900 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:16.739900 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:16.739900 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:16.739900 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:12:16.739900 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:12:16.739900 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:12:16.739900 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:12:16.741198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:16.741198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:16.741198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:16.741198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:16.741198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:16.741198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:16.741198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:16.741198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:16.741198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:16.741198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:16.741198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:16.741198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:16.741198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:16.741198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:16.741198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:16.741198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:16.741198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:12:16.747014 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:12:16.747014 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:12:16.747014 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:16.747014 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 322/651] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:12:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:16.748029 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67da44c2-1838-4291-87a1-7bf5bc729502 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:16.759080 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:16.775945 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d0388b-48d5-480f-bd0c-38bec97ca185 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:16.825790 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab16285a-4fb2-45ec-8db4-4a3e4878206e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:16.826564 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab16285a-4fb2-45ec-8db4-4a3e4878206e None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:16.826843 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab16285a-4fb2-45ec-8db4-4a3e4878206e None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:16.827761 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab16285a-4fb2-45ec-8db4-4a3e4878206e None admin] PATH_INFO: `/v3/services` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:16.828539 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab16285a-4fb2-45ec-8db4-4a3e4878206e None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:16.828979 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab16285a-4fb2-45ec-8db4-4a3e4878206e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:16.830541 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab16285a-4fb2-45ec-8db4-4a3e4878206e None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:16.843661 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 330/652] 10.222.0.42 () {56 vars in 1114 bytes} [Thu Oct 27 01:12:16 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:16.843661 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203fd1a3-9f65-4553-917b-c599a0cf38c8 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:16.843661 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203fd1a3-9f65-4553-917b-c599a0cf38c8 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:16.843661 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203fd1a3-9f65-4553-917b-c599a0cf38c8 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:16.843661 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 331/653] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:16.846508 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c5f669-a703-42ab-b61f-06e42a889727 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:16.847007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c5f669-a703-42ab-b61f-06e42a889727 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:16.847413 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c5f669-a703-42ab-b61f-06e42a889727 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:16.871163 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67da44c2-1838-4291-87a1-7bf5bc729502 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:16.871509 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-04c5f669-a703-42ab-b61f-06e42a889727 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:16.872724 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-04c5f669-a703-42ab-b61f-06e42a889727 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:16.880255 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:16.913415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bbdfa0b-1293-4c26-a478-6da12ff352d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:16.914400 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbdfa0b-1293-4c26-a478-6da12ff352d6 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:16.914910 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbdfa0b-1293-4c26-a478-6da12ff352d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:16.915522 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbdfa0b-1293-4c26-a478-6da12ff352d6 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:16.916409 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bbdfa0b-1293-4c26-a478-6da12ff352d6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:16.916923 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bbdfa0b-1293-4c26-a478-6da12ff352d6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:16.918544 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bbdfa0b-1293-4c26-a478-6da12ff352d6 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:16.934596 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 323/654] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:12:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:16.943696 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bbdfa0b-1293-4c26-a478-6da12ff352d6 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:16.961636 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 332/655] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:16 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:12:17.058525 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bbdfa0b-1293-4c26-a478-6da12ff352d6 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:17.095731 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48969a1c-2902-4874-8a4f-e798da0e8e06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:17.096368 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48969a1c-2902-4874-8a4f-e798da0e8e06 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:17.096730 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48969a1c-2902-4874-8a4f-e798da0e8e06 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:17.097089 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48969a1c-2902-4874-8a4f-e798da0e8e06 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:17.105283 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48969a1c-2902-4874-8a4f-e798da0e8e06 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:17.106759 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48969a1c-2902-4874-8a4f-e798da0e8e06 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:17.120373 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-48969a1c-2902-4874-8a4f-e798da0e8e06 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 27 01:12:17.120373 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:17.120373 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:17.120373 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:17.120373 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:17.120373 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:17.120373 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:17.120373 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:17.120373 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:17.120373 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:17.120373 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:17.120373 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:17.120373 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 27 01:12:17.120373 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 27 01:12:17.120373 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 27 01:12:17.120373 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 27 01:12:17.121408 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:17.121408 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:17.121408 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 27 01:12:17.121408 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:12:17.121408 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 27 01:12:17.121408 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:12:17.121408 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:17.121408 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:17.121408 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:17.121408 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:17.121408 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:17.121408 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:17.121408 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:17.121408 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:17.121408 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:17.121408 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:17.121408 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:17.121408 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:17.122392 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:17.122392 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:17.122392 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 27 01:12:17.122392 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 27 01:12:17.122392 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 27 01:12:17.122392 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 27 01:12:17.122392 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 27 01:12:17.122392 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 27 01:12:17.122392 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 27 01:12:17.122392 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:17.122392 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 324/656] 10.222.0.42 () {56 vars in 1117 bytes} [Thu Oct 27 01:12:16 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:17.130185 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c5f669-a703-42ab-b61f-06e42a889727 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:17.214048 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:17.233885 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c5f669-a703-42ab-b61f-06e42a889727 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:17.261794 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48969a1c-2902-4874-8a4f-e798da0e8e06 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:17.270832 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa96b27-7f30-4ab3-a1c8-162fdb57f7af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:17.271443 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa96b27-7f30-4ab3-a1c8-162fdb57f7af None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:17.271721 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa96b27-7f30-4ab3-a1c8-162fdb57f7af None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:17.271990 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa96b27-7f30-4ab3-a1c8-162fdb57f7af None admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:17.272823 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa96b27-7f30-4ab3-a1c8-162fdb57f7af None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:17.273155 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa96b27-7f30-4ab3-a1c8-162fdb57f7af None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:17.274677 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa96b27-7f30-4ab3-a1c8-162fdb57f7af None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:17.293840 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:17.294874 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 333/657] 10.222.0.42 () {56 vars in 1149 bytes} [Thu Oct 27 01:12:17 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:17.300690 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa96b27-7f30-4ab3-a1c8-162fdb57f7af None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:17.375132 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48969a1c-2902-4874-8a4f-e798da0e8e06 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:17.412384 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa96b27-7f30-4ab3-a1c8-162fdb57f7af None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:17.417712 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d17a4f-eb06-4a5a-bdba-c5ca533aba31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:17.418606 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d17a4f-eb06-4a5a-bdba-c5ca533aba31 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:17.419105 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d17a4f-eb06-4a5a-bdba-c5ca533aba31 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:17.419578 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d17a4f-eb06-4a5a-bdba-c5ca533aba31 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:17.420912 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7d17a4f-eb06-4a5a-bdba-c5ca533aba31 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:17.422991 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7d17a4f-eb06-4a5a-bdba-c5ca533aba31 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:17.435751 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-a7d17a4f-eb06-4a5a-bdba-c5ca533aba31 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 27 01:12:17.435751 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:17.435751 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 27 01:12:17.435751 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 27 01:12:17.435751 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 27 01:12:17.435751 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 27 01:12:17.435751 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 27 01:12:17.435751 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:17.435751 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 27 01:12:17.435751 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:17.435751 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 27 01:12:17.435751 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:17.435751 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:17.435751 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:17.435751 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:17.435751 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:17.436791 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:17.436791 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:17.436791 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:17.436791 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:17.436791 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:17.436791 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:17.436791 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:17.436791 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 27 01:12:17.436791 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 27 01:12:17.436791 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 27 01:12:17.436791 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 27 01:12:17.436791 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:17.436791 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:17.436791 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:17.436791 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:17.436791 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:17.436791 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:17.437729 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:17.437729 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:17.437729 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:17.437729 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:17.437729 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:17.437729 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:17.437729 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:17.437729 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:17.437729 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:17.437729 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:17.437729 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 27 01:12:17.437729 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:17.437729 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 27 01:12:17.437729 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:17.439155 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 325/658] 10.222.0.42 () {56 vars in 1132 bytes} [Thu Oct 27 01:12:17 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:17.444307 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d17a4f-eb06-4a5a-bdba-c5ca533aba31 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:17.458607 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999639b0-43c5-4a99-a049-476c474c5f07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:17.459248 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999639b0-43c5-4a99-a049-476c474c5f07 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:17.459526 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999639b0-43c5-4a99-a049-476c474c5f07 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:17.459876 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999639b0-43c5-4a99-a049-476c474c5f07 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:17.472893 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:17.477043 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-999639b0-43c5-4a99-a049-476c474c5f07 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:17.479595 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-999639b0-43c5-4a99-a049-476c474c5f07 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:17.499019 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-999639b0-43c5-4a99-a049-476c474c5f07 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:12:17.499019 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:17.499019 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:17.499019 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:17.499019 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:17.499019 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:17.499019 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:17.499019 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:17.499019 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:17.499019 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:17.499019 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:17.499019 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:17.499019 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:12:17.499019 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:12:17.499019 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:12:17.499019 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:12:17.500035 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:17.500035 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:17.500035 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:17.500035 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:17.500035 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:17.500035 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:17.500035 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:17.500035 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:17.500035 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:17.500035 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:17.500035 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:17.500035 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:17.500035 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:17.500035 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:17.500035 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:17.500035 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:17.500035 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:12:17.500989 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:12:17.500989 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:12:17.500989 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:17.505316 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 334/659] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:12:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 27 01:12:17.508023 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999639b0-43c5-4a99-a049-476c474c5f07 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:17.561916 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:17.574138 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d17a4f-eb06-4a5a-bdba-c5ca533aba31 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:17.610346 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339e7c9d-44a7-4d01-b044-0bec90b060a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:17.610954 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339e7c9d-44a7-4d01-b044-0bec90b060a3 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:17.611229 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339e7c9d-44a7-4d01-b044-0bec90b060a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:17.611499 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339e7c9d-44a7-4d01-b044-0bec90b060a3 None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:17.613609 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-339e7c9d-44a7-4d01-b044-0bec90b060a3 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:17.613609 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-339e7c9d-44a7-4d01-b044-0bec90b060a3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:17.614640 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-339e7c9d-44a7-4d01-b044-0bec90b060a3 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:17.619882 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 326/660] 10.222.0.42 () {56 vars in 1126 bytes} [Thu Oct 27 01:12:17 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:12:17.624988 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339e7c9d-44a7-4d01-b044-0bec90b060a3 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:17.655682 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999639b0-43c5-4a99-a049-476c474c5f07 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:17.702830 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f169de5c-2b1f-41e8-b983-219877d43a0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:17.704048 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f169de5c-2b1f-41e8-b983-219877d43a0f None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:17.704513 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f169de5c-2b1f-41e8-b983-219877d43a0f None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:17.705502 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f169de5c-2b1f-41e8-b983-219877d43a0f None admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:17.709748 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f169de5c-2b1f-41e8-b983-219877d43a0f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:17.710224 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f169de5c-2b1f-41e8-b983-219877d43a0f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:17.712237 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f169de5c-2b1f-41e8-b983-219877d43a0f None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:17.730366 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 335/661] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:12:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:12:17.738616 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f169de5c-2b1f-41e8-b983-219877d43a0f None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:17.739269 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339e7c9d-44a7-4d01-b044-0bec90b060a3 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:17.774746 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec487e0-f9d9-4f04-8906-600358fe3aaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:17.775465 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec487e0-f9d9-4f04-8906-600358fe3aaf None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:17.775745 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec487e0-f9d9-4f04-8906-600358fe3aaf None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:17.776015 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec487e0-f9d9-4f04-8906-600358fe3aaf None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:17.776643 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec487e0-f9d9-4f04-8906-600358fe3aaf None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:17.776943 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec487e0-f9d9-4f04-8906-600358fe3aaf None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:17.778354 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec487e0-f9d9-4f04-8906-600358fe3aaf None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:17.788537 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 327/662] 10.222.0.42 () {56 vars in 1126 bytes} [Thu Oct 27 01:12:17 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:12:17.789551 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec487e0-f9d9-4f04-8906-600358fe3aaf None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:17.791958 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:17.810043 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:17.868371 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f169de5c-2b1f-41e8-b983-219877d43a0f None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:17.905294 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc406bad-7509-4bef-bf86-2c455af5ef16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:17.905928 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc406bad-7509-4bef-bf86-2c455af5ef16 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:17.906207 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc406bad-7509-4bef-bf86-2c455af5ef16 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:17.906477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc406bad-7509-4bef-bf86-2c455af5ef16 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:17.908086 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec487e0-f9d9-4f04-8906-600358fe3aaf None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:17.914786 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc406bad-7509-4bef-bf86-2c455af5ef16 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:17.916322 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc406bad-7509-4bef-bf86-2c455af5ef16 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:17.928149 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-cc406bad-7509-4bef-bf86-2c455af5ef16 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:12:17.928149 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:17.928149 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:17.928149 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:17.928149 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:17.928149 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:17.928149 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:17.928149 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:17.928149 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:17.928149 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:17.928149 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:17.928149 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:17.928149 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 27 01:12:17.928149 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 27 01:12:17.928149 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 27 01:12:17.929092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 27 01:12:17.929092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:17.929092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:17.929092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:17.929092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:17.929092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:17.929092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:17.929092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:17.929092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:17.929092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:17.929092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:17.929092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:17.929092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:17.929092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:17.929092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:17.929092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:17.929092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:17.929981 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 27 01:12:17.929981 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 27 01:12:17.929981 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 27 01:12:17.929981 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 27 01:12:17.929981 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 27 01:12:17.929981 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 27 01:12:17.929981 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:12:17.929981 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:17.929981 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 336/663] 10.222.0.42 () {56 vars in 1111 bytes} [Thu Oct 27 01:12:17 2022] GET /identity/v3/projects/service => generated 88 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:17.935073 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc406bad-7509-4bef-bf86-2c455af5ef16 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:17.954549 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d1477e-731f-4f65-b569-fe636906b385 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:17.955705 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d1477e-731f-4f65-b569-fe636906b385 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:17.957453 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d1477e-731f-4f65-b569-fe636906b385 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:17.957453 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d1477e-731f-4f65-b569-fe636906b385 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:17.958223 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d1477e-731f-4f65-b569-fe636906b385 None admin] RBAC: Adding query filter params (interface=public, service_id=2c2755e3105744718195fd25c7a99913, region_id=RegionOne) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:17.958532 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d1477e-731f-4f65-b569-fe636906b385 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:17.959932 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d1477e-731f-4f65-b569-fe636906b385 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:17.974624 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 328/664] 10.222.0.42 () {56 vars in 1251 bytes} [Thu Oct 27 01:12:17 2022] GET /identity/v3/endpoints?service_id=2c2755e3105744718195fd25c7a99913&interface=public®ion_id=RegionOne => generated 194 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:12:17.975511 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d1477e-731f-4f65-b569-fe636906b385 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:18.062946 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:18.080839 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc406bad-7509-4bef-bf86-2c455af5ef16 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:18.095426 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:18.099418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d1477e-731f-4f65-b569-fe636906b385 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:18.118616 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c7cc51-1e63-45d2-9377-39c1ae6a09ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:18.119240 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c7cc51-1e63-45d2-9377-39c1ae6a09ba None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:18.119588 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c7cc51-1e63-45d2-9377-39c1ae6a09ba None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:18.119925 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c7cc51-1e63-45d2-9377-39c1ae6a09ba None admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:18.120686 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c7cc51-1e63-45d2-9377-39c1ae6a09ba None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:18.121075 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c7cc51-1e63-45d2-9377-39c1ae6a09ba None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:18.122642 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c7cc51-1e63-45d2-9377-39c1ae6a09ba None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:18.136129 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 337/665] 10.222.0.42 () {56 vars in 1148 bytes} [Thu Oct 27 01:12:17 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:18.144388 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c7cc51-1e63-45d2-9377-39c1ae6a09ba None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:18.149438 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eaef384-3588-4540-a65b-1a5b7b36e0ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:18.150153 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eaef384-3588-4540-a65b-1a5b7b36e0ef None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:18.151021 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eaef384-3588-4540-a65b-1a5b7b36e0ef None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:18.151395 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eaef384-3588-4540-a65b-1a5b7b36e0ef None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:18.152117 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eaef384-3588-4540-a65b-1a5b7b36e0ef None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:18.154489 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eaef384-3588-4540-a65b-1a5b7b36e0ef None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:18.165167 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 329/666] 10.222.0.42 () {56 vars in 1087 bytes} [Thu Oct 27 01:12:17 2022] GET /identity/v3/services => generated 1029 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 27 01:12:18.264936 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c7cc51-1e63-45d2-9377-39c1ae6a09ba None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:18.308283 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100acd41-995d-4f1d-bd04-1bf9dd5e6e82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:18.308984 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100acd41-995d-4f1d-bd04-1bf9dd5e6e82 None admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:18.309273 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100acd41-995d-4f1d-bd04-1bf9dd5e6e82 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:18.309546 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100acd41-995d-4f1d-bd04-1bf9dd5e6e82 None admin] PATH_INFO: `/v3/projects/23aad848d47c4d27ad4c138ff869e40f/users/7a3f8b5eb09f4ba29f4bd343bd91f6a2/roles/04d145c26220455b969c13d2d24e84e8` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:18.311980 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-100acd41-995d-4f1d-bd04-1bf9dd5e6e82 None admin] RBAC: Authorizing `identity:create_grant(project_id=23aad848d47c4d27ad4c138ff869e40f, user_id=7a3f8b5eb09f4ba29f4bd343bd91f6a2, role_id=04d145c26220455b969c13d2d24e84e8)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:18.313562 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-100acd41-995d-4f1d-bd04-1bf9dd5e6e82 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:18.335295 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 338/667] 10.222.0.42 () {58 vars in 1439 bytes} [Thu Oct 27 01:12:18 2022] PUT /identity/v3/projects/23aad848d47c4d27ad4c138ff869e40f/users/7a3f8b5eb09f4ba29f4bd343bd91f6a2/roles/04d145c26220455b969c13d2d24e84e8 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:12:18.416966 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239c3e88-aab3-46fd-a927-7d294575ea16 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:18.417523 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239c3e88-aab3-46fd-a927-7d294575ea16 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:18.417523 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239c3e88-aab3-46fd-a927-7d294575ea16 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:18.418867 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 330/668] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:18.425467 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57bd0bc9-59f1-4d3e-8bcd-b6f201bdd5b6 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:18.426359 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57bd0bc9-59f1-4d3e-8bcd-b6f201bdd5b6 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:18.426359 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57bd0bc9-59f1-4d3e-8bcd-b6f201bdd5b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:18.452119 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-57bd0bc9-59f1-4d3e-8bcd-b6f201bdd5b6 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:18.453645 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-57bd0bc9-59f1-4d3e-8bcd-b6f201bdd5b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:18.485577 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:18.537952 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 339/669] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:18 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:12:18.807907 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239c3e88-aab3-46fd-a927-7d294575ea16 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:18.912740 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239c3e88-aab3-46fd-a927-7d294575ea16 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:18.947930 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc26fc8-5156-466d-967c-2505af7f2073 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:18.948915 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc26fc8-5156-466d-967c-2505af7f2073 None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:18.949423 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc26fc8-5156-466d-967c-2505af7f2073 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:18.949890 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc26fc8-5156-466d-967c-2505af7f2073 None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:18.950760 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfc26fc8-5156-466d-967c-2505af7f2073 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:18.952321 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfc26fc8-5156-466d-967c-2505af7f2073 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:18.977244 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 331/670] 10.222.0.42 () {60 vars in 1141 bytes} [Thu Oct 27 01:12:18 2022] POST /identity/v3/services => generated 241 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:12:19.244843 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f336f3bc-56eb-4bb0-9644-a412f1b7d06d None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:19.245248 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f336f3bc-56eb-4bb0-9644-a412f1b7d06d None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:19.245468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f336f3bc-56eb-4bb0-9644-a412f1b7d06d None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:19.247065 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 340/671] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:19.252332 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c6f7c1-5b54-4f8e-8a66-0601a1c132a2 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:19.252630 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c6f7c1-5b54-4f8e-8a66-0601a1c132a2 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:19.253157 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c6f7c1-5b54-4f8e-8a66-0601a1c132a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:19.266318 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:19.272576 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-e2c6f7c1-5b54-4f8e-8a66-0601a1c132a2 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:19.273811 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e2c6f7c1-5b54-4f8e-8a66-0601a1c132a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:19.353022 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 332/672] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:19 2022] POST /identity/v3/auth/tokens => generated 1373 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:12:19.626629 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f336f3bc-56eb-4bb0-9644-a412f1b7d06d None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:19.734448 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f336f3bc-56eb-4bb0-9644-a412f1b7d06d None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:19.770716 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8c1187-d963-4437-b59d-47c29af5c814 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:19.771688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8c1187-d963-4437-b59d-47c29af5c814 None admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:19.772202 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8c1187-d963-4437-b59d-47c29af5c814 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:19.772686 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8c1187-d963-4437-b59d-47c29af5c814 None admin] PATH_INFO: `/v3/services` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:19.773563 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e8c1187-d963-4437-b59d-47c29af5c814 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:19.775131 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e8c1187-d963-4437-b59d-47c29af5c814 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:19.793960 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 341/673] 10.222.0.42 () {60 vars in 1141 bytes} [Thu Oct 27 01:12:19 2022] POST /identity/v3/services => generated 251 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:12:20.486823 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4b7354-6cb3-49f3-a79b-f4e269b0e444 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:20.487713 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4b7354-6cb3-49f3-a79b-f4e269b0e444 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:20.487713 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4b7354-6cb3-49f3-a79b-f4e269b0e444 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:20.489210 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 333/674] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:20.494853 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56161e3a-a8df-45e5-8343-6f05de551648 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:20.495496 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56161e3a-a8df-45e5-8343-6f05de551648 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:20.495966 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56161e3a-a8df-45e5-8343-6f05de551648 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:20.518253 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-56161e3a-a8df-45e5-8343-6f05de551648 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:20.521176 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-56161e3a-a8df-45e5-8343-6f05de551648 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:20.540216 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:20.599947 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 342/675] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:20 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:12:20.628835 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35873760-0894-4621-a764-6b47265631bd None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:20.629156 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35873760-0894-4621-a764-6b47265631bd None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:20.629439 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35873760-0894-4621-a764-6b47265631bd None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:20.631023 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 334/676] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:20.636313 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bda72b-5b04-480c-ab47-58f762c17963 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:20.636781 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bda72b-5b04-480c-ab47-58f762c17963 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:20.636877 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bda72b-5b04-480c-ab47-58f762c17963 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:20.655841 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-96bda72b-5b04-480c-ab47-58f762c17963 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:20.657023 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-96bda72b-5b04-480c-ab47-58f762c17963 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:20.727429 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 343/677] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:20 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:12:20.887125 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35873760-0894-4621-a764-6b47265631bd None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:20.996794 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35873760-0894-4621-a764-6b47265631bd None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:21.013719 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96bda72b-5b04-480c-ab47-58f762c17963 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:21.034531 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758efc67-e0dc-44eb-b366-03271118075e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:21.035162 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758efc67-e0dc-44eb-b366-03271118075e None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:21.035437 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758efc67-e0dc-44eb-b366-03271118075e None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:21.035708 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758efc67-e0dc-44eb-b366-03271118075e None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:21.036294 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-758efc67-e0dc-44eb-b366-03271118075e None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:21.037716 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-758efc67-e0dc-44eb-b366-03271118075e None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:21.041365 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:21.047170 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-758efc67-e0dc-44eb-b366-03271118075e None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 27 01:12:21.047170 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:21.047170 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 27 01:12:21.047170 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 27 01:12:21.047170 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 27 01:12:21.047170 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 27 01:12:21.047170 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 27 01:12:21.047170 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:21.047170 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 27 01:12:21.047170 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:21.047170 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 27 01:12:21.047170 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:21.047170 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:21.047170 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:21.047170 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:21.047170 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:21.049188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:21.049188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:21.049188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:21.049188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:21.049188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:21.049188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:21.049188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:21.049188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 27 01:12:21.049188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 27 01:12:21.049188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 27 01:12:21.049188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 27 01:12:21.049188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:21.049188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:21.049188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:21.049188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:21.049188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:21.049188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:21.050345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:21.050345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:21.050345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:21.050345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:21.050345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:21.050345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:21.050345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:21.050345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:21.050345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:21.050345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:21.050345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 27 01:12:21.050345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:21.050345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 27 01:12:21.050345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:21.050345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 335/678] 10.222.0.42 () {56 vars in 1132 bytes} [Thu Oct 27 01:12:20 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:21.058048 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758efc67-e0dc-44eb-b366-03271118075e None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:21.100221 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:21.124569 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96bda72b-5b04-480c-ab47-58f762c17963 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:21.168241 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e537ebb-9b50-47e9-b37d-fc2808cf6264 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:21.168990 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e537ebb-9b50-47e9-b37d-fc2808cf6264 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:21.169359 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e537ebb-9b50-47e9-b37d-fc2808cf6264 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:21.169705 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e537ebb-9b50-47e9-b37d-fc2808cf6264 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:21.180728 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e537ebb-9b50-47e9-b37d-fc2808cf6264 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:12:21.185941 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-2e537ebb-9b50-47e9-b37d-fc2808cf6264 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 27 01:12:21.185941 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:21.185941 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:21.185941 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:21.185941 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:21.185941 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:21.185941 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:21.185941 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:21.185941 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:21.185941 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:21.185941 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:21.185941 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:21.185941 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:12:21.185941 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:12:21.185941 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:12:21.185941 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:12:21.187081 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:12:21.187081 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:12:21.187081 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:12:21.187081 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:12:21.187081 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:21.187081 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:21.187081 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:21.187081 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:21.187081 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:21.187081 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:21.187081 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:21.187081 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:21.187081 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:21.187081 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:21.187081 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:21.187081 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:21.188519 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:21.188519 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:21.188519 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:21.188519 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:21.188519 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:12:21.188519 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:12:21.188519 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:12:21.188519 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:12:21.188519 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:12:21.188519 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:12:21.188519 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 27 01:12:21.188519 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:21.189346 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 344/679] 10.222.0.42 () {56 vars in 1120 bytes} [Thu Oct 27 01:12:21 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:21.194659 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758efc67-e0dc-44eb-b366-03271118075e None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:21.195552 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e537ebb-9b50-47e9-b37d-fc2808cf6264 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:21.240186 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab48f396-26ef-4220-9415-46deab8ebaf8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:21.241340 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab48f396-26ef-4220-9415-46deab8ebaf8 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:21.241739 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab48f396-26ef-4220-9415-46deab8ebaf8 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:21.242010 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab48f396-26ef-4220-9415-46deab8ebaf8 None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:21.242697 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab48f396-26ef-4220-9415-46deab8ebaf8 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:21.242997 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab48f396-26ef-4220-9415-46deab8ebaf8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:21.244521 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab48f396-26ef-4220-9415-46deab8ebaf8 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:21.254204 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 336/680] 10.222.0.42 () {56 vars in 1126 bytes} [Thu Oct 27 01:12:21 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:21.260393 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab48f396-26ef-4220-9415-46deab8ebaf8 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:21.326956 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:21.328110 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e537ebb-9b50-47e9-b37d-fc2808cf6264 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:21.364680 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe096efa-6d09-4ceb-9714-2d0793f03a03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:21.365247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe096efa-6d09-4ceb-9714-2d0793f03a03 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:21.365521 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe096efa-6d09-4ceb-9714-2d0793f03a03 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:21.365786 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe096efa-6d09-4ceb-9714-2d0793f03a03 None admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:21.366435 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe096efa-6d09-4ceb-9714-2d0793f03a03 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:21.366731 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe096efa-6d09-4ceb-9714-2d0793f03a03 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:21.368083 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe096efa-6d09-4ceb-9714-2d0793f03a03 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:21.372374 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab48f396-26ef-4220-9415-46deab8ebaf8 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:21.373647 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:21.376947 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 345/681] 10.222.0.42 () {56 vars in 1115 bytes} [Thu Oct 27 01:12:21 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:12:21.378641 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2cfff6-2e4e-466c-a368-216020860f17 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:21.378968 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2cfff6-2e4e-466c-a368-216020860f17 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:21.379244 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2cfff6-2e4e-466c-a368-216020860f17 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:21.381097 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 346/682] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:21.384200 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e2cfff6-2e4e-466c-a368-216020860f17 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:21.416297 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2799395a-cb56-4455-9a8c-030d3d0d4f31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:21.416916 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2799395a-cb56-4455-9a8c-030d3d0d4f31 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:21.417199 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2799395a-cb56-4455-9a8c-030d3d0d4f31 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:21.417469 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2799395a-cb56-4455-9a8c-030d3d0d4f31 None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:21.418091 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2799395a-cb56-4455-9a8c-030d3d0d4f31 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:21.418389 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2799395a-cb56-4455-9a8c-030d3d0d4f31 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:21.419758 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2799395a-cb56-4455-9a8c-030d3d0d4f31 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:21.429699 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 337/683] 10.222.0.42 () {56 vars in 1126 bytes} [Thu Oct 27 01:12:21 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:21.429699 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b958a0-c61e-4ba2-9ecf-e8cd3ba3b5c9 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:21.429699 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b958a0-c61e-4ba2-9ecf-e8cd3ba3b5c9 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:21.429699 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b958a0-c61e-4ba2-9ecf-e8cd3ba3b5c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:21.444925 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-11b958a0-c61e-4ba2-9ecf-e8cd3ba3b5c9 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:21.446147 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-11b958a0-c61e-4ba2-9ecf-e8cd3ba3b5c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:21.497266 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e2cfff6-2e4e-466c-a368-216020860f17 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:21.515281 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 338/684] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:21 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:12:21.516712 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b958a0-c61e-4ba2-9ecf-e8cd3ba3b5c9 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:21.543515 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db253e8a-8fe3-4138-8cba-843e7533a18d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:21.544413 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db253e8a-8fe3-4138-8cba-843e7533a18d None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:21.544960 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db253e8a-8fe3-4138-8cba-843e7533a18d None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:21.545263 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db253e8a-8fe3-4138-8cba-843e7533a18d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:21.557740 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db253e8a-8fe3-4138-8cba-843e7533a18d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:21.559402 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db253e8a-8fe3-4138-8cba-843e7533a18d None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:21.574800 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-db253e8a-8fe3-4138-8cba-843e7533a18d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:12:21.574800 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:21.574800 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:21.574800 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:21.574800 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:21.574800 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:21.574800 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:21.574800 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:21.574800 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:21.574800 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:21.574800 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:21.574800 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:21.574800 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:12:21.574800 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:12:21.574800 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:12:21.574800 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:12:21.575793 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:21.575793 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:21.575793 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:21.575793 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:21.575793 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:21.575793 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:21.575793 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:21.575793 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:21.575793 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:21.575793 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:21.575793 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:21.575793 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:21.575793 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:21.575793 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:21.575793 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:21.575793 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:21.575793 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:12:21.576686 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:12:21.576686 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:12:21.576686 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:21.580753 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 347/685] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:12:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 27 01:12:21.582489 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db253e8a-8fe3-4138-8cba-843e7533a18d None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:21.586313 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:21.647312 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:21.653469 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b958a0-c61e-4ba2-9ecf-e8cd3ba3b5c9 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:21.702951 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4f9029-0556-40a6-bdac-0f5fe8f39018 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:21.703556 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4f9029-0556-40a6-bdac-0f5fe8f39018 None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:21.703829 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4f9029-0556-40a6-bdac-0f5fe8f39018 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:21.704092 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4f9029-0556-40a6-bdac-0f5fe8f39018 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:21.704754 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab4f9029-0556-40a6-bdac-0f5fe8f39018 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:21.706817 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab4f9029-0556-40a6-bdac-0f5fe8f39018 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:21.708286 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db253e8a-8fe3-4138-8cba-843e7533a18d None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:21.728390 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 339/686] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:12:21 2022] POST /identity/v3/endpoints => generated 347 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:12:21.758696 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8ce6b7-2786-40b5-b9ea-955851e22c6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:21.759318 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8ce6b7-2786-40b5-b9ea-955851e22c6e None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:21.759644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8ce6b7-2786-40b5-b9ea-955851e22c6e None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:21.759916 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8ce6b7-2786-40b5-b9ea-955851e22c6e None admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:21.760569 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e8ce6b7-2786-40b5-b9ea-955851e22c6e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:21.760865 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e8ce6b7-2786-40b5-b9ea-955851e22c6e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:21.762311 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e8ce6b7-2786-40b5-b9ea-955851e22c6e None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:21.776771 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 348/687] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:12:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:12:21.782223 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4f9029-0556-40a6-bdac-0f5fe8f39018 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:21.831161 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8ce6b7-2786-40b5-b9ea-955851e22c6e None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:21.905992 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4f9029-0556-40a6-bdac-0f5fe8f39018 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:21.922170 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:21.956883 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e8281c-602c-4f9c-9acc-dd8fbfa370ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:21.957652 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e8281c-602c-4f9c-9acc-dd8fbfa370ff None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:21.957935 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e8281c-602c-4f9c-9acc-dd8fbfa370ff None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:21.958214 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e8281c-602c-4f9c-9acc-dd8fbfa370ff None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:21.964520 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8ce6b7-2786-40b5-b9ea-955851e22c6e None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:21.967071 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e8281c-602c-4f9c-9acc-dd8fbfa370ff None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:21.968449 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e8281c-602c-4f9c-9acc-dd8fbfa370ff None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:21.976880 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:21.982326 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-a9e8281c-602c-4f9c-9acc-dd8fbfa370ff None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 27 01:12:21.982326 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:21.982326 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:21.982326 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:21.982326 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:21.982326 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:21.982326 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:21.982326 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:21.982326 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:21.982326 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:21.982326 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:21.982326 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:21.982326 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 27 01:12:21.982326 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 27 01:12:21.982326 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 27 01:12:21.982326 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 27 01:12:21.983348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:21.983348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:21.983348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 27 01:12:21.983348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:12:21.983348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 27 01:12:21.983348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:12:21.983348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:21.983348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:21.983348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:21.983348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:21.983348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:21.983348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:21.983348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:21.983348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:21.983348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:21.983348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:21.983348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:21.983348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:21.984287 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:21.984287 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:21.984287 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 27 01:12:21.984287 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 27 01:12:21.984287 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 27 01:12:21.984287 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 27 01:12:21.984287 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 27 01:12:21.984287 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 27 01:12:21.984287 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 27 01:12:21.984287 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:21.988656 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 340/688] 10.222.0.42 () {56 vars in 1117 bytes} [Thu Oct 27 01:12:21 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:21.989897 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e8281c-602c-4f9c-9acc-dd8fbfa370ff None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:22.013563 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb06dd5-4641-42e1-a4ad-39d78c4064ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:22.014193 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb06dd5-4641-42e1-a4ad-39d78c4064ae None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:22.014469 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb06dd5-4641-42e1-a4ad-39d78c4064ae None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:22.014739 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb06dd5-4641-42e1-a4ad-39d78c4064ae None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:22.015318 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bb06dd5-4641-42e1-a4ad-39d78c4064ae None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:22.016674 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bb06dd5-4641-42e1-a4ad-39d78c4064ae None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:22.029373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-6bb06dd5-4641-42e1-a4ad-39d78c4064ae None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 27 01:12:22.029373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:22.029373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 27 01:12:22.029373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 27 01:12:22.029373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 27 01:12:22.029373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 27 01:12:22.029373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 27 01:12:22.029373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:22.029373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 27 01:12:22.029373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:22.029373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 27 01:12:22.029373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:22.029373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:22.029373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:22.029373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:22.029373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:22.029373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:22.030436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:22.030436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:22.030436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:22.030436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:22.030436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:22.030436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:22.030436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 27 01:12:22.030436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 27 01:12:22.030436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 27 01:12:22.030436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 27 01:12:22.030436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:22.030436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:22.030436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:22.030436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:22.030436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:22.030436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:22.031298 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:22.031298 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:22.031298 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:22.031298 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:22.031298 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:22.031298 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:22.031298 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:22.031298 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:22.031298 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:22.031298 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:22.031298 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 27 01:12:22.031298 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:22.031298 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 27 01:12:22.031298 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:22.033518 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 349/689] 10.222.0.42 () {56 vars in 1126 bytes} [Thu Oct 27 01:12:21 2022] GET /identity/v3/services/object-store => generated 93 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:22.040990 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb06dd5-4641-42e1-a4ad-39d78c4064ae None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:22.122674 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e8281c-602c-4f9c-9acc-dd8fbfa370ff None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:22.152654 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb06dd5-4641-42e1-a4ad-39d78c4064ae None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:22.158703 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e01ab128-fa46-41ea-9b1f-f7a467c28851 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:22.159393 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e01ab128-fa46-41ea-9b1f-f7a467c28851 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:22.159733 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e01ab128-fa46-41ea-9b1f-f7a467c28851 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:22.160074 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e01ab128-fa46-41ea-9b1f-f7a467c28851 None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:22.160872 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e01ab128-fa46-41ea-9b1f-f7a467c28851 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:22.161239 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e01ab128-fa46-41ea-9b1f-f7a467c28851 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:22.162658 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e01ab128-fa46-41ea-9b1f-f7a467c28851 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:22.180851 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 341/690] 10.222.0.42 () {56 vars in 1149 bytes} [Thu Oct 27 01:12:21 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:12:22.182523 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5df831-2478-48a4-883d-c8fc845bb75f None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:22.182804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5df831-2478-48a4-883d-c8fc845bb75f None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:22.183073 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5df831-2478-48a4-883d-c8fc845bb75f None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:22.184579 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 342/691] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:22.189873 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e712979-fc32-4385-8f2f-a1f02efc69c0 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:22.190523 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e712979-fc32-4385-8f2f-a1f02efc69c0 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:22.190874 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e712979-fc32-4385-8f2f-a1f02efc69c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:22.195273 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c2956c-dd2b-4555-8b59-aa486a19b6df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:22.195882 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c2956c-dd2b-4555-8b59-aa486a19b6df None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:22.196163 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c2956c-dd2b-4555-8b59-aa486a19b6df None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:22.196435 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c2956c-dd2b-4555-8b59-aa486a19b6df None admin] PATH_INFO: `/v3/services` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:22.197081 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c2956c-dd2b-4555-8b59-aa486a19b6df None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:22.197456 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c2956c-dd2b-4555-8b59-aa486a19b6df None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:22.198822 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c2956c-dd2b-4555-8b59-aa486a19b6df None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:22.203974 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 350/692] 10.222.0.42 () {56 vars in 1122 bytes} [Thu Oct 27 01:12:22 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:22.204632 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c2956c-dd2b-4555-8b59-aa486a19b6df None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:22.209239 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-4e712979-fc32-4385-8f2f-a1f02efc69c0 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:22.210417 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-4e712979-fc32-4385-8f2f-a1f02efc69c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:22.233952 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:22.243228 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:22.286340 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 343/693] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:22 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:12:22.293951 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e712979-fc32-4385-8f2f-a1f02efc69c0 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:22.316861 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c2956c-dd2b-4555-8b59-aa486a19b6df None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:22.351685 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d237d2-292e-4e77-b746-ddf8692d89af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:22.352352 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d237d2-292e-4e77-b746-ddf8692d89af None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:22.352814 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d237d2-292e-4e77-b746-ddf8692d89af None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:22.352921 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d237d2-292e-4e77-b746-ddf8692d89af None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:22.425646 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e712979-fc32-4385-8f2f-a1f02efc69c0 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:22.434144 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d237d2-292e-4e77-b746-ddf8692d89af None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:22.436508 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d237d2-292e-4e77-b746-ddf8692d89af None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:22.460693 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a4a91a3-13c3-4c5a-b505-11e316eaae1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:22.461236 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4a91a3-13c3-4c5a-b505-11e316eaae1c None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:22.461516 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4a91a3-13c3-4c5a-b505-11e316eaae1c None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:22.461788 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4a91a3-13c3-4c5a-b505-11e316eaae1c None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:22.462415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a4a91a3-13c3-4c5a-b505-11e316eaae1c None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:22.462717 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a4a91a3-13c3-4c5a-b505-11e316eaae1c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:22.464138 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a4a91a3-13c3-4c5a-b505-11e316eaae1c None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:22.469345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 344/694] 10.222.0.42 () {56 vars in 1122 bytes} [Thu Oct 27 01:12:22 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:22.474937 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a4a91a3-13c3-4c5a-b505-11e316eaae1c None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:22.483137 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:22.500303 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:22.576108 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 351/695] 10.222.0.42 () {58 vars in 1282 bytes} [Thu Oct 27 01:12:22 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 27 01:12:22.576108 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d237d2-292e-4e77-b746-ddf8692d89af None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:22.587247 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a4a91a3-13c3-4c5a-b505-11e316eaae1c None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:22.623359 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a183b290-11d1-496f-a695-bfb9bdd6e541 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:22.623967 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a183b290-11d1-496f-a695-bfb9bdd6e541 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:22.624245 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a183b290-11d1-496f-a695-bfb9bdd6e541 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:22.624516 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a183b290-11d1-496f-a695-bfb9bdd6e541 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:22.625243 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a183b290-11d1-496f-a695-bfb9bdd6e541 None admin] RBAC: Adding query filter params (interface=public, service_id=dc6c6ef7216b48f1a85abaf803e41e2f, region_id=RegionOne) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:22.625549 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a183b290-11d1-496f-a695-bfb9bdd6e541 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:22.626906 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a183b290-11d1-496f-a695-bfb9bdd6e541 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:22.632649 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 345/696] 10.222.0.42 () {56 vars in 1251 bytes} [Thu Oct 27 01:12:22 2022] GET /identity/v3/endpoints?service_id=dc6c6ef7216b48f1a85abaf803e41e2f&interface=public®ion_id=RegionOne => generated 194 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:22.633448 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a183b290-11d1-496f-a695-bfb9bdd6e541 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:22.689842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d237d2-292e-4e77-b746-ddf8692d89af None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:22.723033 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:22.726804 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a95520-c3cd-452b-be5e-d0f523b93f45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:22.727435 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a95520-c3cd-452b-be5e-d0f523b93f45 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:22.727984 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a95520-c3cd-452b-be5e-d0f523b93f45 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:22.728310 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a95520-c3cd-452b-be5e-d0f523b93f45 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:22.729034 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a95520-c3cd-452b-be5e-d0f523b93f45 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:22.730814 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a95520-c3cd-452b-be5e-d0f523b93f45 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:22.733748 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:22.739624 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a183b290-11d1-496f-a695-bfb9bdd6e541 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:22.740857 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-c9a95520-c3cd-452b-be5e-d0f523b93f45 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 27 01:12:22.740857 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:22.740857 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 27 01:12:22.740857 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 27 01:12:22.740857 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 27 01:12:22.740857 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 27 01:12:22.740857 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 27 01:12:22.740857 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:22.740857 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 27 01:12:22.740857 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:22.740857 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 27 01:12:22.740857 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:22.740857 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:22.740857 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:22.740857 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:22.740857 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:22.740857 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:22.741890 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:22.741890 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:22.741890 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:22.741890 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:22.741890 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:22.741890 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:22.741890 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 27 01:12:22.741890 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 27 01:12:22.741890 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 27 01:12:22.741890 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 27 01:12:22.741890 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:22.741890 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:22.741890 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:22.741890 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:22.741890 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:22.741890 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:22.748231 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:22.748231 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:22.748231 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:22.748231 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:22.748231 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:22.748231 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:22.748231 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:22.748231 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:22.748231 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:22.748231 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:22.748231 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 27 01:12:22.748231 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:22.748231 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 27 01:12:22.748231 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:22.748231 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 352/697] 10.222.0.42 () {56 vars in 1114 bytes} [Thu Oct 27 01:12:22 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:22.749050 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a95520-c3cd-452b-be5e-d0f523b93f45 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:22.788080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c75a50-b9b1-4ae2-ba0d-1ff8ab8d7761 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:22.788706 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c75a50-b9b1-4ae2-ba0d-1ff8ab8d7761 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:22.788983 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c75a50-b9b1-4ae2-ba0d-1ff8ab8d7761 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:22.789280 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c75a50-b9b1-4ae2-ba0d-1ff8ab8d7761 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:22.799799 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c75a50-b9b1-4ae2-ba0d-1ff8ab8d7761 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:22.801305 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c75a50-b9b1-4ae2-ba0d-1ff8ab8d7761 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:22.818391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-b9c75a50-b9b1-4ae2-ba0d-1ff8ab8d7761 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:12:22.818391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:22.818391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:22.818391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:22.818391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:22.818391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:22.818391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:22.818391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:22.818391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:22.818391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:22.818391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:22.818391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:22.818391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:12:22.818391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:12:22.818391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:12:22.818391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:12:22.819371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:22.819371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:22.819371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:22.819371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:22.819371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:22.819371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:22.819371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:22.819371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:22.819371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:22.819371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:22.819371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:22.819371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:22.819371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:22.819371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:22.819371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:22.819371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:22.819371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:12:22.820268 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:12:22.820268 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:12:22.820268 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:22.824348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 346/698] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:12:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:22.825234 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c75a50-b9b1-4ae2-ba0d-1ff8ab8d7761 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:22.857608 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a95520-c3cd-452b-be5e-d0f523b93f45 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:22.893009 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065417e2-4a21-4760-a6d1-dc370a3ffdb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:22.894161 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065417e2-4a21-4760-a6d1-dc370a3ffdb4 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:22.894443 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065417e2-4a21-4760-a6d1-dc370a3ffdb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:22.894709 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065417e2-4a21-4760-a6d1-dc370a3ffdb4 None admin] PATH_INFO: `/v3/services` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:22.895306 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-065417e2-4a21-4760-a6d1-dc370a3ffdb4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:22.896876 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-065417e2-4a21-4760-a6d1-dc370a3ffdb4 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:22.902827 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 353/699] 10.222.0.42 () {56 vars in 1087 bytes} [Thu Oct 27 01:12:22 2022] GET /identity/v3/services => generated 1497 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:12:22.902827 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065417e2-4a21-4760-a6d1-dc370a3ffdb4 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:22.945987 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c75a50-b9b1-4ae2-ba0d-1ff8ab8d7761 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:22.986079 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea06bb7-8c88-48ee-a5fa-58ce210e0a8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:22.986695 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea06bb7-8c88-48ee-a5fa-58ce210e0a8b None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:22.987490 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea06bb7-8c88-48ee-a5fa-58ce210e0a8b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:22.987777 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea06bb7-8c88-48ee-a5fa-58ce210e0a8b None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:22.988419 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea06bb7-8c88-48ee-a5fa-58ce210e0a8b None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:22.988718 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea06bb7-8c88-48ee-a5fa-58ce210e0a8b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:22.990165 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea06bb7-8c88-48ee-a5fa-58ce210e0a8b None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:22.996071 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 347/700] 10.222.0.42 () {56 vars in 1114 bytes} [Thu Oct 27 01:12:22 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:23.001984 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea06bb7-8c88-48ee-a5fa-58ce210e0a8b None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:23.005592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:23.012892 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:23.069764 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065417e2-4a21-4760-a6d1-dc370a3ffdb4 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:23.116640 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a4408e-d6c7-487b-a929-717a58fb809f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:23.117386 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a4408e-d6c7-487b-a929-717a58fb809f None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:23.117663 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a4408e-d6c7-487b-a929-717a58fb809f None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:23.117934 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a4408e-d6c7-487b-a929-717a58fb809f None admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:23.118574 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a4408e-d6c7-487b-a929-717a58fb809f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:23.119173 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a4408e-d6c7-487b-a929-717a58fb809f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:23.120355 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a4408e-d6c7-487b-a929-717a58fb809f None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:23.134685 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 354/701] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:12:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:23.135137 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea06bb7-8c88-48ee-a5fa-58ce210e0a8b None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:23.141430 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a4408e-d6c7-487b-a929-717a58fb809f None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:23.173340 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dec6da9-39f0-43ea-8751-feb08da6e3f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:23.173943 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dec6da9-39f0-43ea-8751-feb08da6e3f6 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:23.174214 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dec6da9-39f0-43ea-8751-feb08da6e3f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:23.174987 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dec6da9-39f0-43ea-8751-feb08da6e3f6 None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:23.175767 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dec6da9-39f0-43ea-8751-feb08da6e3f6 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:23.176102 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dec6da9-39f0-43ea-8751-feb08da6e3f6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:23.177803 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dec6da9-39f0-43ea-8751-feb08da6e3f6 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:23.184600 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 348/702] 10.222.0.42 () {56 vars in 1114 bytes} [Thu Oct 27 01:12:23 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:23.190476 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dec6da9-39f0-43ea-8751-feb08da6e3f6 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:23.268563 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a4408e-d6c7-487b-a929-717a58fb809f None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:23.274197 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:23.295365 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:23.308953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863dc41c-b60a-468c-aa3b-d8a417009302 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:23.309702 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863dc41c-b60a-468c-aa3b-d8a417009302 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:23.310156 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863dc41c-b60a-468c-aa3b-d8a417009302 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:23.310264 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863dc41c-b60a-468c-aa3b-d8a417009302 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:23.317749 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-863dc41c-b60a-468c-aa3b-d8a417009302 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:23.319233 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-863dc41c-b60a-468c-aa3b-d8a417009302 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:23.320924 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dec6da9-39f0-43ea-8751-feb08da6e3f6 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:23.330565 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-863dc41c-b60a-468c-aa3b-d8a417009302 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:12:23.330565 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:23.330565 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:23.330565 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:23.330565 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:23.330565 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:23.330565 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:23.330565 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:23.330565 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:23.330565 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:23.330565 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:23.330565 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:23.330565 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 27 01:12:23.330565 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 27 01:12:23.330565 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 27 01:12:23.332055 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 27 01:12:23.332055 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:23.332055 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:23.332055 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:23.332055 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:23.332055 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:23.332055 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:23.332055 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:23.332055 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:23.332055 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:23.332055 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:23.332055 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:23.332055 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:23.332055 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:23.332055 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:23.332055 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:23.332055 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:23.332938 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 27 01:12:23.332938 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 27 01:12:23.332938 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 27 01:12:23.332938 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 27 01:12:23.332938 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 27 01:12:23.332938 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 27 01:12:23.332938 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:12:23.332938 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:23.332938 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 355/703] 10.222.0.42 () {56 vars in 1111 bytes} [Thu Oct 27 01:12:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 27 01:12:23.337955 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863dc41c-b60a-468c-aa3b-d8a417009302 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:23.363978 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a524d21c-fbd4-4dfb-90f3-2337f3468dcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:23.364633 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a524d21c-fbd4-4dfb-90f3-2337f3468dcb None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:23.364911 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a524d21c-fbd4-4dfb-90f3-2337f3468dcb None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:23.365264 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a524d21c-fbd4-4dfb-90f3-2337f3468dcb None admin] PATH_INFO: `/v3/endpoints` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:23.365966 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a524d21c-fbd4-4dfb-90f3-2337f3468dcb None admin] RBAC: Adding query filter params (interface=public, service_id=989b6fd5dd174ff5b93165447ab7529c, region_id=RegionOne) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:23.366265 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a524d21c-fbd4-4dfb-90f3-2337f3468dcb None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:23.367688 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a524d21c-fbd4-4dfb-90f3-2337f3468dcb None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:23.377305 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 349/704] 10.222.0.42 () {56 vars in 1251 bytes} [Thu Oct 27 01:12:23 2022] GET /identity/v3/endpoints?service_id=989b6fd5dd174ff5b93165447ab7529c&interface=public®ion_id=RegionOne => generated 194 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:23.379804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a524d21c-fbd4-4dfb-90f3-2337f3468dcb None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:23.476504 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863dc41c-b60a-468c-aa3b-d8a417009302 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:23.494237 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a524d21c-fbd4-4dfb-90f3-2337f3468dcb None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:23.525732 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208375e6-b8d6-4d6b-9b91-aced4b20569d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:23.526707 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:23.527970 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208375e6-b8d6-4d6b-9b91-aced4b20569d None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:23.528592 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208375e6-b8d6-4d6b-9b91-aced4b20569d None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:23.529344 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208375e6-b8d6-4d6b-9b91-aced4b20569d None admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:23.530238 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-208375e6-b8d6-4d6b-9b91-aced4b20569d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:23.530741 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-208375e6-b8d6-4d6b-9b91-aced4b20569d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:23.532525 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-208375e6-b8d6-4d6b-9b91-aced4b20569d None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:23.534609 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f27820b-036d-4a61-b8c9-af025056d7ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:23.535729 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f27820b-036d-4a61-b8c9-af025056d7ad None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:23.536490 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f27820b-036d-4a61-b8c9-af025056d7ad None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:23.536882 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f27820b-036d-4a61-b8c9-af025056d7ad None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:23.537637 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f27820b-036d-4a61-b8c9-af025056d7ad None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:23.539349 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f27820b-036d-4a61-b8c9-af025056d7ad None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:23.548948 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 356/705] 10.222.0.42 () {56 vars in 1148 bytes} [Thu Oct 27 01:12:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:12:23.549497 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 350/706] 10.222.0.42 () {56 vars in 1087 bytes} [Thu Oct 27 01:12:23 2022] GET /identity/v3/services => generated 1497 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 27 01:12:23.560291 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208375e6-b8d6-4d6b-9b91-aced4b20569d None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:23.616652 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:23.679629 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208375e6-b8d6-4d6b-9b91-aced4b20569d None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:23.721691 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5babc8a8-f499-4c75-9c93-aa4aa0574be1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:23.722925 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5babc8a8-f499-4c75-9c93-aa4aa0574be1 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:23.722925 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5babc8a8-f499-4c75-9c93-aa4aa0574be1 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:23.722925 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5babc8a8-f499-4c75-9c93-aa4aa0574be1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:23.723567 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5babc8a8-f499-4c75-9c93-aa4aa0574be1 None admin] RBAC: Adding query filter params (role.id=04d145c26220455b969c13d2d24e84e8, scope.project.id=23aad848d47c4d27ad4c138ff869e40f, user.id=7a3f8b5eb09f4ba29f4bd343bd91f6a2) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:23.723833 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5babc8a8-f499-4c75-9c93-aa4aa0574be1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:23.725284 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5babc8a8-f499-4c75-9c93-aa4aa0574be1 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:23.732136 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 357/707] 10.222.0.42 () {56 vars in 1374 bytes} [Thu Oct 27 01:12:23 2022] GET /identity/v3/role_assignments?user.id=7a3f8b5eb09f4ba29f4bd343bd91f6a2&scope.project.id=23aad848d47c4d27ad4c138ff869e40f&role.id=04d145c26220455b969c13d2d24e84e8 => generated 608 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:24.053092 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7743f8-3a93-44da-be43-c55e26953e4e None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:24.053610 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7743f8-3a93-44da-be43-c55e26953e4e None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:24.053717 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7743f8-3a93-44da-be43-c55e26953e4e None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:24.055258 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 351/708] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:24.060808 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f791c17-dd62-4d58-9b84-210fd5a9c98e None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:24.061120 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f791c17-dd62-4d58-9b84-210fd5a9c98e None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:24.061395 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f791c17-dd62-4d58-9b84-210fd5a9c98e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:24.080442 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-5f791c17-dd62-4d58-9b84-210fd5a9c98e None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:24.081643 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-5f791c17-dd62-4d58-9b84-210fd5a9c98e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:24.151456 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 358/709] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:24 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:12:24.421188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc7743f8-3a93-44da-be43-c55e26953e4e None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:24.528589 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc7743f8-3a93-44da-be43-c55e26953e4e None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:24.564970 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e1d307-d91f-4f62-93ef-a6199823f4d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:24.565966 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e1d307-d91f-4f62-93ef-a6199823f4d2 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:24.566134 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e1d307-d91f-4f62-93ef-a6199823f4d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:24.566287 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e1d307-d91f-4f62-93ef-a6199823f4d2 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:24.566885 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e1d307-d91f-4f62-93ef-a6199823f4d2 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:24.568356 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e1d307-d91f-4f62-93ef-a6199823f4d2 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:24.577308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-d7e1d307-d91f-4f62-93ef-a6199823f4d2 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 27 01:12:24.577308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:24.577308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 27 01:12:24.577308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 27 01:12:24.577308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 27 01:12:24.577308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 27 01:12:24.577308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 27 01:12:24.577308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:24.577308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 27 01:12:24.577308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:24.577308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 27 01:12:24.577308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:24.577308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:24.577308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:24.577308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:24.577308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:24.577308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:24.579647 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:24.579647 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:24.579647 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:24.579647 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:24.579647 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:24.579647 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:24.579647 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 27 01:12:24.579647 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 27 01:12:24.579647 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 27 01:12:24.579647 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 27 01:12:24.579647 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:24.579647 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:24.579647 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:24.579647 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:24.579647 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:24.579647 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:24.582766 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:24.582766 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:24.582766 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:24.582766 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:24.582766 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:24.582766 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:24.582766 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:24.582766 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:24.582766 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:24.582766 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:24.582766 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 27 01:12:24.582766 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:24.582766 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 27 01:12:24.582766 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:24.582766 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 352/710] 10.222.0.42 () {56 vars in 1111 bytes} [Thu Oct 27 01:12:24 2022] GET /identity/v3/services/compute => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:24.589555 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f791c17-dd62-4d58-9b84-210fd5a9c98e None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:24.626597 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:24.690833 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f791c17-dd62-4d58-9b84-210fd5a9c98e None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:24.729365 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7abc8792-8429-4b91-a5e1-8b8cd4dfe941 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:24.729984 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abc8792-8429-4b91-a5e1-8b8cd4dfe941 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:24.730228 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abc8792-8429-4b91-a5e1-8b8cd4dfe941 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:24.730501 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abc8792-8429-4b91-a5e1-8b8cd4dfe941 None admin] PATH_INFO: `/v3/services` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:24.731124 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7abc8792-8429-4b91-a5e1-8b8cd4dfe941 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:24.731422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7abc8792-8429-4b91-a5e1-8b8cd4dfe941 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:24.732783 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7abc8792-8429-4b91-a5e1-8b8cd4dfe941 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:24.737931 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 359/711] 10.222.0.42 () {56 vars in 1112 bytes} [Thu Oct 27 01:12:24 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:24.743525 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e1d307-d91f-4f62-93ef-a6199823f4d2 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:24.822035 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:24.866606 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e1d307-d91f-4f62-93ef-a6199823f4d2 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:24.902512 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94897d1-0539-4d13-9ce3-d7d81de1ffe0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:24.903116 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94897d1-0539-4d13-9ce3-d7d81de1ffe0 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:24.903392 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94897d1-0539-4d13-9ce3-d7d81de1ffe0 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:24.903676 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94897d1-0539-4d13-9ce3-d7d81de1ffe0 None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:24.904348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a94897d1-0539-4d13-9ce3-d7d81de1ffe0 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:24.904651 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a94897d1-0539-4d13-9ce3-d7d81de1ffe0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:24.906086 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a94897d1-0539-4d13-9ce3-d7d81de1ffe0 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:24.911569 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 353/712] 10.222.0.42 () {56 vars in 1112 bytes} [Thu Oct 27 01:12:24 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:25.257409 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890de54a-aba6-4f8e-9f5e-2f7296219b60 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:25.257747 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890de54a-aba6-4f8e-9f5e-2f7296219b60 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:25.258066 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890de54a-aba6-4f8e-9f5e-2f7296219b60 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:25.259666 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 360/713] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:25.264946 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6534c4e4-29b4-4089-9c75-71aea201822e None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:25.265433 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6534c4e4-29b4-4089-9c75-71aea201822e None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:25.265513 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6534c4e4-29b4-4089-9c75-71aea201822e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:25.286031 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-6534c4e4-29b4-4089-9c75-71aea201822e None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:25.287210 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-6534c4e4-29b4-4089-9c75-71aea201822e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:25.364391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 354/714] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:25 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:12:25.640077 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-890de54a-aba6-4f8e-9f5e-2f7296219b60 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:25.749150 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-890de54a-aba6-4f8e-9f5e-2f7296219b60 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:25.750725 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:25.791404 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c8f6d8-cc2b-4450-9845-aee5306b7d39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:25.792355 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c8f6d8-cc2b-4450-9845-aee5306b7d39 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:25.792804 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c8f6d8-cc2b-4450-9845-aee5306b7d39 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:25.793256 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c8f6d8-cc2b-4450-9845-aee5306b7d39 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:25.794025 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90c8f6d8-cc2b-4450-9845-aee5306b7d39 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:25.795650 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90c8f6d8-cc2b-4450-9845-aee5306b7d39 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:25.806002 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-90c8f6d8-cc2b-4450-9845-aee5306b7d39 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 27 01:12:25.806002 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:25.806002 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 27 01:12:25.806002 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 27 01:12:25.806002 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 27 01:12:25.806002 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 27 01:12:25.806002 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 27 01:12:25.806002 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:25.806002 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 27 01:12:25.806002 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:25.806002 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 27 01:12:25.806002 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:25.806002 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:25.806002 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:25.806002 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:25.806002 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:25.806002 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:25.807625 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:25.807625 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:25.807625 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:25.807625 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:25.807625 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:25.807625 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:25.807625 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 27 01:12:25.807625 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 27 01:12:25.807625 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 27 01:12:25.807625 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 27 01:12:25.807625 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:25.807625 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:25.807625 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:25.807625 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:25.807625 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:25.807625 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:25.808663 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:25.808663 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:25.808663 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:25.808663 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:25.808663 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:25.808663 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:25.808663 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:25.808663 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:25.808663 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:25.808663 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:25.808663 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 27 01:12:25.808663 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:25.808663 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 27 01:12:25.808663 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:25.811240 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 361/715] 10.222.0.42 () {56 vars in 1126 bytes} [Thu Oct 27 01:12:25 2022] GET /identity/v3/services/object-store => generated 93 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:25.818773 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6534c4e4-29b4-4089-9c75-71aea201822e None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:25.885751 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:25.937098 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bed09eb-447d-47c7-8eed-30ca15729b03 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:25.937721 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bed09eb-447d-47c7-8eed-30ca15729b03 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:25.938231 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6534c4e4-29b4-4089-9c75-71aea201822e None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:25.938343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bed09eb-447d-47c7-8eed-30ca15729b03 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:25.940625 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 362/716] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 27 01:12:25.946267 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a5f525-e98a-4e6e-877e-f5369dd82786 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:25.946793 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a5f525-e98a-4e6e-877e-f5369dd82786 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:25.947261 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a5f525-e98a-4e6e-877e-f5369dd82786 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:25.965535 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-c4a5f525-e98a-4e6e-877e-f5369dd82786 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:25.967038 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-c4a5f525-e98a-4e6e-877e-f5369dd82786 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:25.977012 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f19ae0c-4823-431b-9690-388458c9e49d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:25.977761 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f19ae0c-4823-431b-9690-388458c9e49d None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:25.977959 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f19ae0c-4823-431b-9690-388458c9e49d None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:25.978190 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f19ae0c-4823-431b-9690-388458c9e49d None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:25.978879 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f19ae0c-4823-431b-9690-388458c9e49d None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:25.979185 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f19ae0c-4823-431b-9690-388458c9e49d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:25.980646 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f19ae0c-4823-431b-9690-388458c9e49d None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:25.986178 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 355/717] 10.222.0.42 () {56 vars in 1122 bytes} [Thu Oct 27 01:12:25 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:25.989305 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3d5024-3345-4128-99fd-7d6c81a3c631 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:25.989596 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3d5024-3345-4128-99fd-7d6c81a3c631 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:25.989869 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3d5024-3345-4128-99fd-7d6c81a3c631 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:25.991338 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 356/718] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:25.992934 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3d5024-3345-4128-99fd-7d6c81a3c631 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:26.045095 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 363/719] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:25 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 27 01:12:26.045095 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7653bc-ec86-4ba7-8226-eba0d88c8c25 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:26.045095 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7653bc-ec86-4ba7-8226-eba0d88c8c25 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:26.045095 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7653bc-ec86-4ba7-8226-eba0d88c8c25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:26.064620 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-ef7653bc-ec86-4ba7-8226-eba0d88c8c25 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:26.066349 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-ef7653bc-ec86-4ba7-8226-eba0d88c8c25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:26.128871 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3d5024-3345-4128-99fd-7d6c81a3c631 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:26.149462 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 364/720] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:26 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:12:26.150225 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:26.169373 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c011b2f-f39c-4777-8417-b1d5b30f0ccc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:26.170308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c011b2f-f39c-4777-8417-b1d5b30f0ccc None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:26.170785 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c011b2f-f39c-4777-8417-b1d5b30f0ccc None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:26.171278 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c011b2f-f39c-4777-8417-b1d5b30f0ccc None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:26.172105 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c011b2f-f39c-4777-8417-b1d5b30f0ccc None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:26.172604 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c011b2f-f39c-4777-8417-b1d5b30f0ccc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:26.174195 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c011b2f-f39c-4777-8417-b1d5b30f0ccc None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:26.180550 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 357/721] 10.222.0.42 () {56 vars in 1122 bytes} [Thu Oct 27 01:12:25 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:26.186374 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7653bc-ec86-4ba7-8226-eba0d88c8c25 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:26.216120 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:26.296617 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7653bc-ec86-4ba7-8226-eba0d88c8c25 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:26.314315 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c011b2f-f39c-4777-8417-b1d5b30f0ccc None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:26.340763 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a433769-8238-45ad-b775-17781064a29b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:26.341541 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a433769-8238-45ad-b775-17781064a29b None admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:26.341928 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a433769-8238-45ad-b775-17781064a29b None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:26.342200 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a433769-8238-45ad-b775-17781064a29b None admin] PATH_INFO: `/v3/endpoints` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:26.342872 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a433769-8238-45ad-b775-17781064a29b None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:26.344910 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a433769-8238-45ad-b775-17781064a29b None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:26.372263 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 365/722] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:12:26 2022] POST /identity/v3/endpoints => generated 349 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:12:26.433684 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a433769-8238-45ad-b775-17781064a29b None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:26.436742 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c011b2f-f39c-4777-8417-b1d5b30f0ccc None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:26.482496 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb824ce-fad2-46a4-bf0e-6ff486266232 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:26.483417 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb824ce-fad2-46a4-bf0e-6ff486266232 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:26.483912 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb824ce-fad2-46a4-bf0e-6ff486266232 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:26.484333 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb824ce-fad2-46a4-bf0e-6ff486266232 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:26.485085 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb824ce-fad2-46a4-bf0e-6ff486266232 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:26.486610 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb824ce-fad2-46a4-bf0e-6ff486266232 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:26.496592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-afb824ce-fad2-46a4-bf0e-6ff486266232 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 27 01:12:26.496592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:26.496592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 27 01:12:26.496592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 27 01:12:26.496592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 27 01:12:26.496592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 27 01:12:26.496592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 27 01:12:26.496592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:26.496592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 27 01:12:26.496592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:26.496592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 27 01:12:26.496592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:26.496592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:26.496592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:26.496592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:26.496592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:26.496592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:26.497861 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:26.497861 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:26.497861 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:26.497861 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:26.497861 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:26.497861 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:26.497861 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 27 01:12:26.497861 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 27 01:12:26.497861 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 27 01:12:26.497861 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 27 01:12:26.497861 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:26.497861 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:26.497861 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:26.497861 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:26.497861 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:26.497861 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:26.498945 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:26.498945 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:26.498945 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:26.498945 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:26.498945 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:26.498945 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:26.498945 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:26.498945 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:26.498945 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:26.498945 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:26.498945 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 27 01:12:26.498945 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:26.498945 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 27 01:12:26.498945 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:26.501520 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 358/723] 10.222.0.42 () {56 vars in 1114 bytes} [Thu Oct 27 01:12:26 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 27 01:12:26.508601 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb824ce-fad2-46a4-bf0e-6ff486266232 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:26.535510 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:26.558290 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:26.558497 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a433769-8238-45ad-b775-17781064a29b None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:26.599536 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6092066c-1791-4945-8c79-58da2fcbea56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:26.599536 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6092066c-1791-4945-8c79-58da2fcbea56 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:26.599536 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6092066c-1791-4945-8c79-58da2fcbea56 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:26.599885 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6092066c-1791-4945-8c79-58da2fcbea56 None admin] PATH_INFO: `/v3/services/image` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:26.600768 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6092066c-1791-4945-8c79-58da2fcbea56 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:26.602855 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6092066c-1791-4945-8c79-58da2fcbea56 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:26.614607 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-6092066c-1791-4945-8c79-58da2fcbea56 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 27 01:12:26.614607 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:26.614607 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 27 01:12:26.614607 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 27 01:12:26.614607 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 27 01:12:26.614607 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 27 01:12:26.614607 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 27 01:12:26.614607 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:26.614607 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 27 01:12:26.614607 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:26.614607 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 27 01:12:26.614607 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:26.614607 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:26.614607 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:26.614607 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:26.614607 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:26.614607 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:26.615636 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:26.615636 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:26.615636 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:26.615636 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:26.615636 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:26.615636 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:26.615636 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 27 01:12:26.615636 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 27 01:12:26.615636 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 27 01:12:26.615636 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 27 01:12:26.615636 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:26.615636 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:26.615636 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:26.615636 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:26.615636 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:26.615636 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:26.616479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:26.616479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:26.616479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:26.616479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:26.616479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:26.616479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:26.616479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:26.616479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:26.616479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:26.616479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:26.616479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 27 01:12:26.616479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:26.616479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 27 01:12:26.616479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:26.616479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 366/724] 10.222.0.42 () {56 vars in 1105 bytes} [Thu Oct 27 01:12:26 2022] GET /identity/v3/services/image => generated 86 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:26.622214 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6092066c-1791-4945-8c79-58da2fcbea56 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:26.626783 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb824ce-fad2-46a4-bf0e-6ff486266232 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:26.662396 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ad1d74-8f56-408b-abfb-8a73fe280a29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:26.663002 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ad1d74-8f56-408b-abfb-8a73fe280a29 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:26.663282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ad1d74-8f56-408b-abfb-8a73fe280a29 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:26.663556 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ad1d74-8f56-408b-abfb-8a73fe280a29 None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:26.664191 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19ad1d74-8f56-408b-abfb-8a73fe280a29 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:26.664496 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19ad1d74-8f56-408b-abfb-8a73fe280a29 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:26.665893 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19ad1d74-8f56-408b-abfb-8a73fe280a29 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:26.671083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 359/725] 10.222.0.42 () {56 vars in 1114 bytes} [Thu Oct 27 01:12:26 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:12:26.677271 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ad1d74-8f56-408b-abfb-8a73fe280a29 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:26.735200 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6092066c-1791-4945-8c79-58da2fcbea56 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:26.775139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0c0381-8d37-429c-91bc-fa9463d2828f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:26.775801 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0c0381-8d37-429c-91bc-fa9463d2828f None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:26.776026 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0c0381-8d37-429c-91bc-fa9463d2828f None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:26.776255 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0c0381-8d37-429c-91bc-fa9463d2828f None admin] PATH_INFO: `/v3/services` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:26.777020 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf0c0381-8d37-429c-91bc-fa9463d2828f None admin] RBAC: Adding query filter params (name=image) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:26.777364 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf0c0381-8d37-429c-91bc-fa9463d2828f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:26.778872 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf0c0381-8d37-429c-91bc-fa9463d2828f None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:26.785031 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 367/726] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:12:26 2022] GET /identity/v3/services?name=image => generated 122 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:12:26.790583 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0c0381-8d37-429c-91bc-fa9463d2828f None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:26.797248 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ad1d74-8f56-408b-abfb-8a73fe280a29 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:26.801374 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:26.808683 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:26.842990 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9b7b5d-1bd2-4e43-8eef-59bee8ad1673 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:26.843980 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9b7b5d-1bd2-4e43-8eef-59bee8ad1673 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:26.844493 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9b7b5d-1bd2-4e43-8eef-59bee8ad1673 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:26.844973 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9b7b5d-1bd2-4e43-8eef-59bee8ad1673 None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:26.845877 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab9b7b5d-1bd2-4e43-8eef-59bee8ad1673 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:26.846468 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab9b7b5d-1bd2-4e43-8eef-59bee8ad1673 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:26.848055 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab9b7b5d-1bd2-4e43-8eef-59bee8ad1673 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:26.854204 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 360/727] 10.222.0.42 () {56 vars in 1114 bytes} [Thu Oct 27 01:12:26 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:26.859629 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9b7b5d-1bd2-4e43-8eef-59bee8ad1673 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:26.927009 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0c0381-8d37-429c-91bc-fa9463d2828f None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:26.972398 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f4dd94-edf4-4bd3-b780-227fe3dd5b70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:26.973790 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f4dd94-edf4-4bd3-b780-227fe3dd5b70 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:26.974450 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f4dd94-edf4-4bd3-b780-227fe3dd5b70 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:26.974925 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f4dd94-edf4-4bd3-b780-227fe3dd5b70 None admin] PATH_INFO: `/v3/services` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:26.976368 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f4dd94-edf4-4bd3-b780-227fe3dd5b70 None admin] RBAC: Adding query filter params (type=image) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:26.976877 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f4dd94-edf4-4bd3-b780-227fe3dd5b70 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:26.979318 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f4dd94-edf4-4bd3-b780-227fe3dd5b70 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:26.984326 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9b7b5d-1bd2-4e43-8eef-59bee8ad1673 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:26.985631 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 368/728] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:12:26 2022] GET /identity/v3/services?type=image => generated 122 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:12:27.029512 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b88a087-e7e1-4e1a-adbe-e0c8f9e2567f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:27.030143 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b88a087-e7e1-4e1a-adbe-e0c8f9e2567f None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:27.030418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b88a087-e7e1-4e1a-adbe-e0c8f9e2567f None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:27.030683 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b88a087-e7e1-4e1a-adbe-e0c8f9e2567f None admin] PATH_INFO: `/v3/endpoints` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:27.031438 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b88a087-e7e1-4e1a-adbe-e0c8f9e2567f None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:27.032916 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b88a087-e7e1-4e1a-adbe-e0c8f9e2567f None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:27.060216 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 361/729] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:12:26 2022] POST /identity/v3/endpoints => generated 346 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:12:27.143842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22df2368-55b5-4bd6-94d7-b5c655c33c10 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:27.144179 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22df2368-55b5-4bd6-94d7-b5c655c33c10 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:27.144391 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22df2368-55b5-4bd6-94d7-b5c655c33c10 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:27.146357 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 369/730] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:27.152410 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ce4e8f-b73e-4760-ad97-37eaa93dd810 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:27.152741 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ce4e8f-b73e-4760-ad97-37eaa93dd810 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:27.152952 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ce4e8f-b73e-4760-ad97-37eaa93dd810 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:27.173788 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-43ce4e8f-b73e-4760-ad97-37eaa93dd810 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:27.175059 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-43ce4e8f-b73e-4760-ad97-37eaa93dd810 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:27.196990 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:27.260543 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 362/731] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:27 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:12:27.538042 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22df2368-55b5-4bd6-94d7-b5c655c33c10 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:27.618073 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:27.645633 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22df2368-55b5-4bd6-94d7-b5c655c33c10 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:27.680889 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9955121-8acf-4d3a-984f-c16422e786f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:27.681761 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9955121-8acf-4d3a-984f-c16422e786f0 None admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:27.682032 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9955121-8acf-4d3a-984f-c16422e786f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:27.682297 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9955121-8acf-4d3a-984f-c16422e786f0 None admin] PATH_INFO: `/v3/services` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:27.682948 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9955121-8acf-4d3a-984f-c16422e786f0 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:27.684305 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9955121-8acf-4d3a-984f-c16422e786f0 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:27.700584 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 370/732] 10.222.0.42 () {60 vars in 1141 bytes} [Thu Oct 27 01:12:27 2022] POST /identity/v3/services => generated 242 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:12:28.693008 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb303a4c-734e-45a3-b373-6a2edfe085fd None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:28.693551 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb303a4c-734e-45a3-b373-6a2edfe085fd None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:28.693551 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb303a4c-734e-45a3-b373-6a2edfe085fd None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:28.695178 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 363/733] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:28.700721 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a503b0-a05f-4a39-8ce9-1eb269f145d3 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:28.701591 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a503b0-a05f-4a39-8ce9-1eb269f145d3 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:28.702072 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a503b0-a05f-4a39-8ce9-1eb269f145d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:28.725090 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-c6a503b0-a05f-4a39-8ce9-1eb269f145d3 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:28.726513 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-c6a503b0-a05f-4a39-8ce9-1eb269f145d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:28.803264 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 371/734] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:28 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:12:29.071967 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb303a4c-734e-45a3-b373-6a2edfe085fd None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:29.183429 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb303a4c-734e-45a3-b373-6a2edfe085fd None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:29.223571 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b229ad4c-4d58-46b6-9bc5-bc8d1c5c70bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:29.224648 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b229ad4c-4d58-46b6-9bc5-bc8d1c5c70bf None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:29.225166 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b229ad4c-4d58-46b6-9bc5-bc8d1c5c70bf None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:29.225631 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b229ad4c-4d58-46b6-9bc5-bc8d1c5c70bf None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:29.226417 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b229ad4c-4d58-46b6-9bc5-bc8d1c5c70bf None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:29.228071 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b229ad4c-4d58-46b6-9bc5-bc8d1c5c70bf None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:29.234305 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54000d10-a8b4-4cb7-8158-768a6cd6c792 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:29.234588 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54000d10-a8b4-4cb7-8158-768a6cd6c792 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:29.234862 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54000d10-a8b4-4cb7-8158-768a6cd6c792 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:29.236439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 372/735] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 27 01:12:29.239194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-b229ad4c-4d58-46b6-9bc5-bc8d1c5c70bf None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 27 01:12:29.239194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:29.239194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 27 01:12:29.239194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 27 01:12:29.239194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 27 01:12:29.239194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 27 01:12:29.239194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 27 01:12:29.239194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:29.239194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 27 01:12:29.239194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:29.239194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 27 01:12:29.239194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:29.239194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:29.239194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:29.239194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:29.239194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:29.239194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:29.240150 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:29.240150 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:29.240150 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:29.240150 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:29.240150 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:29.240150 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:29.240150 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 27 01:12:29.240150 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 27 01:12:29.240150 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 27 01:12:29.240150 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 27 01:12:29.240150 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:29.240150 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:29.240150 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:29.240150 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:29.240150 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:29.240150 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:29.241010 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:29.241010 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:29.241010 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:29.241010 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:29.241010 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:29.241010 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:29.241010 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:29.241010 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:29.241010 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:29.241010 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:29.241010 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 27 01:12:29.241010 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:29.241010 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 27 01:12:29.241010 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:29.241817 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6f511a-024d-4557-8eb5-24d97ba5d80c None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:29.241817 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6f511a-024d-4557-8eb5-24d97ba5d80c None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:29.241817 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6f511a-024d-4557-8eb5-24d97ba5d80c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:29.243564 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 364/736] 10.222.0.42 () {56 vars in 1126 bytes} [Thu Oct 27 01:12:29 2022] GET /identity/v3/services/object-store => generated 93 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:29.249153 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b229ad4c-4d58-46b6-9bc5-bc8d1c5c70bf None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:29.263155 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-7a6f511a-024d-4557-8eb5-24d97ba5d80c None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:29.264351 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-7a6f511a-024d-4557-8eb5-24d97ba5d80c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:29.272057 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:29.334809 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:29.350691 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 373/737] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:29 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 27 01:12:29.352004 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a1dad5-1cc4-4c3d-9174-b54812cea596 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:29.352289 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a1dad5-1cc4-4c3d-9174-b54812cea596 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:29.352561 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a1dad5-1cc4-4c3d-9174-b54812cea596 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:29.353990 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 374/738] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:29.359267 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3775f5a7-d4e7-4494-96ba-300a0dec038b None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:29.359565 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3775f5a7-d4e7-4494-96ba-300a0dec038b None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:29.359829 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3775f5a7-d4e7-4494-96ba-300a0dec038b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:29.375507 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b229ad4c-4d58-46b6-9bc5-bc8d1c5c70bf None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:29.382482 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-3775f5a7-d4e7-4494-96ba-300a0dec038b None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:29.383679 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-3775f5a7-d4e7-4494-96ba-300a0dec038b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:29.420702 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3899ab33-aed9-44b3-ba12-0221e5c044fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:29.422392 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3899ab33-aed9-44b3-ba12-0221e5c044fc None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:29.422876 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3899ab33-aed9-44b3-ba12-0221e5c044fc None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:29.423387 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3899ab33-aed9-44b3-ba12-0221e5c044fc None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:29.424251 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3899ab33-aed9-44b3-ba12-0221e5c044fc None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:29.424757 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3899ab33-aed9-44b3-ba12-0221e5c044fc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:29.427524 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3899ab33-aed9-44b3-ba12-0221e5c044fc None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:29.433681 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 365/739] 10.222.0.42 () {56 vars in 1122 bytes} [Thu Oct 27 01:12:29 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:12:29.439285 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3899ab33-aed9-44b3-ba12-0221e5c044fc None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:29.468485 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 375/740] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:29 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:12:29.530578 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:29.554641 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3899ab33-aed9-44b3-ba12-0221e5c044fc None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:29.594562 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a615399-3ae4-49bc-8f90-beab41eb02c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:29.595100 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a615399-3ae4-49bc-8f90-beab41eb02c1 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:29.595379 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a615399-3ae4-49bc-8f90-beab41eb02c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:29.595655 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a615399-3ae4-49bc-8f90-beab41eb02c1 None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:29.596341 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a615399-3ae4-49bc-8f90-beab41eb02c1 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:29.596654 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a615399-3ae4-49bc-8f90-beab41eb02c1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:29.598062 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a615399-3ae4-49bc-8f90-beab41eb02c1 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:29.604979 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 366/741] 10.222.0.42 () {56 vars in 1122 bytes} [Thu Oct 27 01:12:29 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:12:29.610100 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3775f5a7-d4e7-4494-96ba-300a0dec038b None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:29.620420 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a615399-3ae4-49bc-8f90-beab41eb02c1 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:29.730017 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a615399-3ae4-49bc-8f90-beab41eb02c1 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:29.739125 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3775f5a7-d4e7-4494-96ba-300a0dec038b None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:29.765380 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:29.771141 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa809a9-2230-4376-b95e-1c6dcc3ee6a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:29.772085 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa809a9-2230-4376-b95e-1c6dcc3ee6a9 None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:29.772273 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa809a9-2230-4376-b95e-1c6dcc3ee6a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:29.772645 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa809a9-2230-4376-b95e-1c6dcc3ee6a9 None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:29.773501 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa809a9-2230-4376-b95e-1c6dcc3ee6a9 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:29.774928 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa809a9-2230-4376-b95e-1c6dcc3ee6a9 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:29.785777 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26744f12-1797-4b9f-8b84-6cbe02b548c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:29.786354 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26744f12-1797-4b9f-8b84-6cbe02b548c3 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:29.786646 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26744f12-1797-4b9f-8b84-6cbe02b548c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:29.787021 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26744f12-1797-4b9f-8b84-6cbe02b548c3 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:29.787763 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26744f12-1797-4b9f-8b84-6cbe02b548c3 None admin] RBAC: Adding query filter params (interface=admin, service_id=dc6c6ef7216b48f1a85abaf803e41e2f, region_id=RegionOne) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:29.788075 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26744f12-1797-4b9f-8b84-6cbe02b548c3 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:29.789456 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26744f12-1797-4b9f-8b84-6cbe02b548c3 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:29.793943 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 367/742] 10.222.0.42 () {60 vars in 1141 bytes} [Thu Oct 27 01:12:29 2022] POST /identity/v3/services => generated 242 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:12:29.795111 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 376/743] 10.222.0.42 () {56 vars in 1249 bytes} [Thu Oct 27 01:12:29 2022] GET /identity/v3/endpoints?service_id=dc6c6ef7216b48f1a85abaf803e41e2f&interface=admin®ion_id=RegionOne => generated 193 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:29.803870 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa809a9-2230-4376-b95e-1c6dcc3ee6a9 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:29.804062 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26744f12-1797-4b9f-8b84-6cbe02b548c3 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:29.810579 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:29.945930 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26744f12-1797-4b9f-8b84-6cbe02b548c3 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:29.956876 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa809a9-2230-4376-b95e-1c6dcc3ee6a9 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:29.995415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78096af-d8cd-4ba4-9aca-9376ba50e959 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:29.995919 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af70dc3-d31c-4eb1-885b-12790f91d6e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:29.996037 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78096af-d8cd-4ba4-9aca-9376ba50e959 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:29.996291 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78096af-d8cd-4ba4-9aca-9376ba50e959 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:29.997115 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af70dc3-d31c-4eb1-885b-12790f91d6e2 None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:29.997262 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78096af-d8cd-4ba4-9aca-9376ba50e959 None admin] PATH_INFO: `/v3/services` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:29.997351 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af70dc3-d31c-4eb1-885b-12790f91d6e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:29.997351 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af70dc3-d31c-4eb1-885b-12790f91d6e2 None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:29.997493 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a78096af-d8cd-4ba4-9aca-9376ba50e959 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:29.997727 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af70dc3-d31c-4eb1-885b-12790f91d6e2 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:29.998591 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a78096af-d8cd-4ba4-9aca-9376ba50e959 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:30.000612 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af70dc3-d31c-4eb1-885b-12790f91d6e2 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:30.001953 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: WARNING keystone.server.flask.common [None req-2af70dc3-d31c-4eb1-885b-12790f91d6e2 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Oct 27 01:12:30.006234 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 377/744] 10.222.0.42 () {56 vars in 1087 bytes} [Thu Oct 27 01:12:29 2022] GET /identity/v3/services => generated 1957 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:12:30.009289 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06b33cd-b300-4187-8d46-b8d1cc3104f0 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:30.009289 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06b33cd-b300-4187-8d46-b8d1cc3104f0 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:30.009289 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06b33cd-b300-4187-8d46-b8d1cc3104f0 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:30.010255 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 378/745] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:30.016130 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2fa966-529d-49e1-bbb1-515e1bf28735 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:30.016429 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2fa966-529d-49e1-bbb1-515e1bf28735 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:30.016674 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2fa966-529d-49e1-bbb1-515e1bf28735 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:30.022759 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.sql.core [None req-2af70dc3-d31c-4eb1-885b-12790f91d6e2 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 27 01:12:30.022759 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 27 01:12:30.022759 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [parameters: {'id': '63ca91efc140492ab4ec8bb6d3a310d6', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 27 01:12:30.022759 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81587) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 27 01:12:30.037960 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-3b2fa966-529d-49e1-bbb1-515e1bf28735 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:30.039177 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-3b2fa966-529d-49e1-bbb1-515e1bf28735 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:30.041110 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-2af70dc3-d31c-4eb1-885b-12790f91d6e2 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 63ca91efc140492ab4ec8bb6d3a310d6.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 63ca91efc140492ab4ec8bb6d3a310d6. Oct 27 01:12:30.041110 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:30.041110 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 27 01:12:30.041110 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 27 01:12:30.041110 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 27 01:12:30.041110 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 27 01:12:30.041110 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 27 01:12:30.041110 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application result = self._query(query) Oct 27 01:12:30.041110 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 27 01:12:30.041110 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application conn.query(q) Oct 27 01:12:30.041110 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 27 01:12:30.041110 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 27 01:12:30.041110 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 27 01:12:30.041958 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application result.read() Oct 27 01:12:30.041958 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 27 01:12:30.041958 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 27 01:12:30.041958 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 27 01:12:30.041958 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 27 01:12:30.041958 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 27 01:12:30.041958 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 27 01:12:30.041958 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 27 01:12:30.041958 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 27 01:12:30.041958 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 27 01:12:30.041958 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:30.041958 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 27 01:12:30.041958 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:30.041958 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:30.041958 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 27 01:12:30.041958 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 27 01:12:30.041958 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 27 01:12:30.041958 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 27 01:12:30.042930 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 27 01:12:30.042930 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application next(self.gen) Oct 27 01:12:30.042930 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1068, in _transaction_scope Oct 27 01:12:30.042930 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application yield resource Oct 27 01:12:30.042930 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 27 01:12:30.042930 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application next(self.gen) Oct 27 01:12:30.042930 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 670, in _session Oct 27 01:12:30.042930 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self.session.rollback() Oct 27 01:12:30.042930 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 27 01:12:30.042930 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self.force_reraise() Oct 27 01:12:30.042930 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 27 01:12:30.042930 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise self.value Oct 27 01:12:30.042930 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 667, in _session Oct 27 01:12:30.042930 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 27 01:12:30.042930 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 695, in _end_session_transaction Oct 27 01:12:30.042930 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application session.commit() Oct 27 01:12:30.042930 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 27 01:12:30.043845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 27 01:12:30.043845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 27 01:12:30.043845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._prepare_impl() Oct 27 01:12:30.043845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 27 01:12:30.043845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self.session.flush() Oct 27 01:12:30.043845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 27 01:12:30.043845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._flush(objects) Oct 27 01:12:30.043845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 27 01:12:30.043845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 27 01:12:30.043845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 27 01:12:30.043845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application compat.raise_( Oct 27 01:12:30.043845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 27 01:12:30.043845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception Oct 27 01:12:30.043845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 27 01:12:30.043845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application flush_context.execute() Oct 27 01:12:30.043845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 27 01:12:30.043845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rec.execute(self) Oct 27 01:12:30.046234 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 27 01:12:30.046234 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 27 01:12:30.046234 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 27 01:12:30.046234 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 27 01:12:30.046234 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 27 01:12:30.046234 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 27 01:12:30.046234 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 27 01:12:30.046234 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 27 01:12:30.046234 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 27 01:12:30.046234 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 27 01:12:30.046234 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 27 01:12:30.046234 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 27 01:12:30.046234 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 27 01:12:30.046234 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 27 01:12:30.046234 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 27 01:12:30.048046 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 27 01:12:30.048046 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 27 01:12:30.048046 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception Oct 27 01:12:30.048046 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 27 01:12:30.048046 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 27 01:12:30.048046 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 27 01:12:30.048046 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 27 01:12:30.048046 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 27 01:12:30.048046 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application result = self._query(query) Oct 27 01:12:30.048046 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 27 01:12:30.048046 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application conn.query(q) Oct 27 01:12:30.048046 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 27 01:12:30.048046 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 27 01:12:30.048046 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 27 01:12:30.048046 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application result.read() Oct 27 01:12:30.048046 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 27 01:12:30.048046 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 27 01:12:30.049415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 27 01:12:30.049415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 27 01:12:30.049415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 27 01:12:30.049415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 27 01:12:30.049415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 27 01:12:30.049415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 27 01:12:30.049415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 27 01:12:30.049415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 27 01:12:30.049415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [parameters: {'id': '63ca91efc140492ab4ec8bb6d3a310d6', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 27 01:12:30.049415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 27 01:12:30.049415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:30.049415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 27 01:12:30.049415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:30.049415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:30.049415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 27 01:12:30.050770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 27 01:12:30.050770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 27 01:12:30.050770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 27 01:12:30.050770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Oct 27 01:12:30.050770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:30.050770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 27 01:12:30.050770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:30.050770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:30.050770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:30.050770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:30.050770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:30.050770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:30.050770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:30.050770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:30.050770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:30.050770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:30.050770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:30.051886 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:30.051886 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Oct 27 01:12:30.051886 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Oct 27 01:12:30.051886 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:30.051886 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:30.051886 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 27 01:12:30.051886 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 27 01:12:30.051886 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 63ca91efc140492ab4ec8bb6d3a310d6. Oct 27 01:12:30.051886 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:30.051886 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 368/746] 10.222.0.42 () {60 vars in 1140 bytes} [Thu Oct 27 01:12:29 2022] POST /identity/v3/projects => generated 262 bytes in 248 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:30.051886 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af70dc3-d31c-4eb1-885b-12790f91d6e2 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:30.109923 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:30.121241 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:30.147376 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 379/747] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:30 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 27 01:12:30.183525 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af70dc3-d31c-4eb1-885b-12790f91d6e2 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:30.222124 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9461ea9-fc0c-463b-883a-48607fe09e74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:30.222862 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9461ea9-fc0c-463b-883a-48607fe09e74 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:30.223235 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9461ea9-fc0c-463b-883a-48607fe09e74 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:30.223605 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9461ea9-fc0c-463b-883a-48607fe09e74 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:30.230632 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9461ea9-fc0c-463b-883a-48607fe09e74 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:30.232374 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9461ea9-fc0c-463b-883a-48607fe09e74 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:30.244269 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-a9461ea9-fc0c-463b-883a-48607fe09e74 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:12:30.244269 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:30.244269 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:30.244269 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:30.244269 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:30.244269 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:30.244269 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:30.244269 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:30.244269 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:30.244269 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:30.244269 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:30.244269 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:30.244269 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 27 01:12:30.244269 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 27 01:12:30.244269 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 27 01:12:30.245428 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 27 01:12:30.245428 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:30.245428 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:30.245428 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:30.245428 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:30.245428 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:30.245428 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:30.245428 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:30.245428 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:30.245428 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:30.245428 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:30.245428 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:30.245428 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:30.245428 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:30.245428 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:30.245428 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:30.245428 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:30.246492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 27 01:12:30.246492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 27 01:12:30.246492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 27 01:12:30.246492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 27 01:12:30.246492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 27 01:12:30.246492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 27 01:12:30.246492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:12:30.246492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:30.246492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 369/748] 10.222.0.42 () {56 vars in 1111 bytes} [Thu Oct 27 01:12:30 2022] GET /identity/v3/projects/service => generated 88 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 27 01:12:30.251717 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2fa966-529d-49e1-bbb1-515e1bf28735 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:30.358264 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2fa966-529d-49e1-bbb1-515e1bf28735 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:30.396177 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72febc95-f443-47cd-8c85-51ee0ba0eefa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:30.396954 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72febc95-f443-47cd-8c85-51ee0ba0eefa None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:30.397377 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72febc95-f443-47cd-8c85-51ee0ba0eefa None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:30.397747 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72febc95-f443-47cd-8c85-51ee0ba0eefa None admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:30.400118 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72febc95-f443-47cd-8c85-51ee0ba0eefa None admin] RBAC: Adding query filter params (name=service) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:30.400530 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72febc95-f443-47cd-8c85-51ee0ba0eefa None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:30.402141 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72febc95-f443-47cd-8c85-51ee0ba0eefa None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:30.420782 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 380/749] 10.222.0.42 () {56 vars in 1112 bytes} [Thu Oct 27 01:12:30 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:30.458982 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9461ea9-fc0c-463b-883a-48607fe09e74 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:30.574703 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9461ea9-fc0c-463b-883a-48607fe09e74 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:30.590152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:30.614811 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b344d974-e6c8-426f-8fc1-82f4c3f52919 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:30.615446 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b344d974-e6c8-426f-8fc1-82f4c3f52919 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:30.615724 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b344d974-e6c8-426f-8fc1-82f4c3f52919 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:30.615996 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b344d974-e6c8-426f-8fc1-82f4c3f52919 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:30.616587 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b344d974-e6c8-426f-8fc1-82f4c3f52919 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:30.618005 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b344d974-e6c8-426f-8fc1-82f4c3f52919 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:30.627119 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-b344d974-e6c8-426f-8fc1-82f4c3f52919 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 27 01:12:30.627119 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:30.627119 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 27 01:12:30.627119 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 27 01:12:30.627119 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 27 01:12:30.627119 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 27 01:12:30.627119 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 27 01:12:30.627119 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:30.627119 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 27 01:12:30.627119 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:30.627119 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 27 01:12:30.627119 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:30.627119 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:30.627119 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:30.627119 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:30.627119 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:30.627119 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:30.628542 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:30.628542 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:30.628542 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:30.628542 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:30.628542 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:30.628542 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:30.628542 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 27 01:12:30.628542 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 27 01:12:30.628542 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 27 01:12:30.628542 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 27 01:12:30.628542 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:30.628542 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:30.628542 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:30.628542 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:30.628542 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:30.628542 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:30.629619 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:30.629619 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:30.629619 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:30.629619 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:30.629619 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:30.629619 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:30.629619 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:30.629619 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:30.629619 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:30.629619 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:30.629619 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 27 01:12:30.629619 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:30.629619 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 27 01:12:30.629619 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:30.633548 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 370/750] 10.222.0.42 () {56 vars in 1111 bytes} [Thu Oct 27 01:12:30 2022] GET /identity/v3/services/compute => generated 88 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:30.634653 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72febc95-f443-47cd-8c85-51ee0ba0eefa None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:30.680961 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:30.750538 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72febc95-f443-47cd-8c85-51ee0ba0eefa None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:30.787679 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2523e3-2f01-4fb2-8937-c11e7d160730 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:30.788272 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2523e3-2f01-4fb2-8937-c11e7d160730 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:30.788552 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2523e3-2f01-4fb2-8937-c11e7d160730 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:30.788824 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2523e3-2f01-4fb2-8937-c11e7d160730 None admin] PATH_INFO: `/v3/services` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:30.789490 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2523e3-2f01-4fb2-8937-c11e7d160730 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:30.789788 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2523e3-2f01-4fb2-8937-c11e7d160730 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:30.791155 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2523e3-2f01-4fb2-8937-c11e7d160730 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:30.801181 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 381/751] 10.222.0.42 () {56 vars in 1112 bytes} [Thu Oct 27 01:12:30 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:30.803778 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b344d974-e6c8-426f-8fc1-82f4c3f52919 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:30.922546 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b344d974-e6c8-426f-8fc1-82f4c3f52919 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:30.957757 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a83d2e3-abac-41c4-bd9c-05db1a5f3848 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:30.958366 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a83d2e3-abac-41c4-bd9c-05db1a5f3848 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:30.958643 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a83d2e3-abac-41c4-bd9c-05db1a5f3848 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:30.958911 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a83d2e3-abac-41c4-bd9c-05db1a5f3848 None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:30.959542 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a83d2e3-abac-41c4-bd9c-05db1a5f3848 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:30.959840 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a83d2e3-abac-41c4-bd9c-05db1a5f3848 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:30.961235 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a83d2e3-abac-41c4-bd9c-05db1a5f3848 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:30.969407 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 371/752] 10.222.0.42 () {56 vars in 1112 bytes} [Thu Oct 27 01:12:30 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:30.975905 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2523e3-2f01-4fb2-8937-c11e7d160730 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:31.087514 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2523e3-2f01-4fb2-8937-c11e7d160730 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:31.089011 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:31.123390 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ce61b2-c0f0-41ee-a9ee-ecd4448c3f11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:31.123996 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ce61b2-c0f0-41ee-a9ee-ecd4448c3f11 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:31.124277 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ce61b2-c0f0-41ee-a9ee-ecd4448c3f11 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:31.124549 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ce61b2-c0f0-41ee-a9ee-ecd4448c3f11 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:31.125273 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ce61b2-c0f0-41ee-a9ee-ecd4448c3f11 None admin] RBAC: Adding query filter params (interface=public, service_id=666d2037c420468d84680ae254c7c8f4, region_id=RegionOne) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:31.125578 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ce61b2-c0f0-41ee-a9ee-ecd4448c3f11 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:31.126943 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ce61b2-c0f0-41ee-a9ee-ecd4448c3f11 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:31.132609 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 382/753] 10.222.0.42 () {56 vars in 1251 bytes} [Thu Oct 27 01:12:30 2022] GET /identity/v3/endpoints?service_id=666d2037c420468d84680ae254c7c8f4&interface=public®ion_id=RegionOne => generated 194 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:31.138154 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a83d2e3-abac-41c4-bd9c-05db1a5f3848 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:31.189506 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:31.252780 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a83d2e3-abac-41c4-bd9c-05db1a5f3848 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:31.288407 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69663b62-79d1-4f3b-9543-2b25d8fffd1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:31.289418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69663b62-79d1-4f3b-9543-2b25d8fffd1e None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:31.289899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69663b62-79d1-4f3b-9543-2b25d8fffd1e None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:31.290388 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69663b62-79d1-4f3b-9543-2b25d8fffd1e None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:31.291202 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69663b62-79d1-4f3b-9543-2b25d8fffd1e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:31.292750 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69663b62-79d1-4f3b-9543-2b25d8fffd1e None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:31.300297 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 372/754] 10.222.0.42 () {56 vars in 1087 bytes} [Thu Oct 27 01:12:31 2022] GET /identity/v3/services => generated 1957 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:12:32.121135 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6921f6-54c9-4766-add6-454193c6c9c1 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:32.122093 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6921f6-54c9-4766-add6-454193c6c9c1 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:32.122581 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6921f6-54c9-4766-add6-454193c6c9c1 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:32.124343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 383/755] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:32.130324 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd5a92f-af28-4891-a888-abf8ec90e99b None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:32.131136 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd5a92f-af28-4891-a888-abf8ec90e99b None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:32.131610 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd5a92f-af28-4891-a888-abf8ec90e99b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:32.151142 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-7fd5a92f-af28-4891-a888-abf8ec90e99b None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:32.152928 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-7fd5a92f-af28-4891-a888-abf8ec90e99b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:32.232180 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 373/756] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:32 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:12:32.296618 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ebb3e3-d98b-44b9-8af0-e114958f381f None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:32.296926 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ebb3e3-d98b-44b9-8af0-e114958f381f None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:32.297204 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ebb3e3-d98b-44b9-8af0-e114958f381f None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:32.298764 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 384/757] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:32.304054 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ba8e1b-985d-4e86-b260-f44fb1f304e8 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:32.304339 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ba8e1b-985d-4e86-b260-f44fb1f304e8 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:32.304612 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ba8e1b-985d-4e86-b260-f44fb1f304e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:32.324101 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-85ba8e1b-985d-4e86-b260-f44fb1f304e8 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:32.325305 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-85ba8e1b-985d-4e86-b260-f44fb1f304e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:32.355614 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:32.395220 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 374/758] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:32 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:12:32.509447 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ebb3e3-d98b-44b9-8af0-e114958f381f None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:32.618101 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ebb3e3-d98b-44b9-8af0-e114958f381f None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:32.654348 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4662fbea-0dd7-4bd3-881e-9a294d936c3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:32.654899 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4662fbea-0dd7-4bd3-881e-9a294d936c3d None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:32.655170 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4662fbea-0dd7-4bd3-881e-9a294d936c3d None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:32.655434 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4662fbea-0dd7-4bd3-881e-9a294d936c3d None admin] PATH_INFO: `/v3/services/image` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:32.656015 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4662fbea-0dd7-4bd3-881e-9a294d936c3d None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:32.657402 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4662fbea-0dd7-4bd3-881e-9a294d936c3d None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:32.666593 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-4662fbea-0dd7-4bd3-881e-9a294d936c3d None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 27 01:12:32.666593 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:32.666593 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 27 01:12:32.666593 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 27 01:12:32.666593 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 27 01:12:32.666593 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 27 01:12:32.666593 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 27 01:12:32.666593 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:32.666593 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 27 01:12:32.666593 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:32.666593 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 27 01:12:32.666593 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:32.666593 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:32.666593 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:32.666593 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:32.666593 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:32.666593 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:32.667588 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:32.667588 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:32.667588 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:32.667588 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:32.667588 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:32.667588 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:32.667588 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 27 01:12:32.667588 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 27 01:12:32.667588 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 27 01:12:32.667588 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 27 01:12:32.667588 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:32.667588 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:32.667588 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:32.667588 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:32.667588 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:32.667588 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:32.668423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:32.668423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:32.668423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:32.668423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:32.668423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:32.668423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:32.668423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:32.668423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:32.668423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:32.668423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:32.668423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 27 01:12:32.668423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:32.668423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 27 01:12:32.668423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:32.669772 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ba8e1b-985d-4e86-b260-f44fb1f304e8 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:32.669849 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 385/759] 10.222.0.42 () {56 vars in 1105 bytes} [Thu Oct 27 01:12:32 2022] GET /identity/v3/services/image => generated 86 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:32.674653 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4662fbea-0dd7-4bd3-881e-9a294d936c3d None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:32.709085 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:32.782312 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4662fbea-0dd7-4bd3-881e-9a294d936c3d None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:32.785957 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ba8e1b-985d-4e86-b260-f44fb1f304e8 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:32.822530 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e195c5d-1749-499c-b72f-472383983d53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:32.823138 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e195c5d-1749-499c-b72f-472383983d53 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:32.823412 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e195c5d-1749-499c-b72f-472383983d53 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:32.823677 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e195c5d-1749-499c-b72f-472383983d53 None admin] PATH_INFO: `/v3/services` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:32.824306 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e195c5d-1749-499c-b72f-472383983d53 None admin] RBAC: Adding query filter params (name=image) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:32.824644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e195c5d-1749-499c-b72f-472383983d53 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:32.825982 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e195c5d-1749-499c-b72f-472383983d53 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:32.826227 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db2d15b-cdae-4e7c-9419-b693d440b450 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:32.826963 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db2d15b-cdae-4e7c-9419-b693d440b450 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:32.827238 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db2d15b-cdae-4e7c-9419-b693d440b450 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:32.827716 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db2d15b-cdae-4e7c-9419-b693d440b450 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:32.828184 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db2d15b-cdae-4e7c-9419-b693d440b450 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:32.829639 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db2d15b-cdae-4e7c-9419-b693d440b450 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:32.831544 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 386/760] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:12:32 2022] GET /identity/v3/services?name=image => generated 122 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:32.833107 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3da0724-62b1-40fe-a867-20611e5c0bfc None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:32.833107 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3da0724-62b1-40fe-a867-20611e5c0bfc None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:32.833107 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3da0724-62b1-40fe-a867-20611e5c0bfc None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:32.834559 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 387/761] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:32 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:32.837194 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3da0724-62b1-40fe-a867-20611e5c0bfc None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:32.840415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-0db2d15b-cdae-4e7c-9419-b693d440b450 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 27 01:12:32.840415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:32.840415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 27 01:12:32.840415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 27 01:12:32.840415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 27 01:12:32.840415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 27 01:12:32.840415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 27 01:12:32.840415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:32.840415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 27 01:12:32.840415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:32.840415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 27 01:12:32.840415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:32.840415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:32.840415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:32.840415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:32.840415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:32.840415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:32.841442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:32.841442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:32.841442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:32.841442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:32.841442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:32.841442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:32.841442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 27 01:12:32.841442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 27 01:12:32.841442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 27 01:12:32.841442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 27 01:12:32.841442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:32.841442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:32.841442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:32.841442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:32.841442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:32.841442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:32.842308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:32.842308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:32.842308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:32.842308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:32.842308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:32.842308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:32.842308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:32.842308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:32.842308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:32.842308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:32.842308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 27 01:12:32.842308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:32.842308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 27 01:12:32.842308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:32.849439 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 375/762] 10.222.0.42 () {56 vars in 1126 bytes} [Thu Oct 27 01:12:32 2022] GET /identity/v3/services/object-store => generated 93 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:32.849439 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3a1f47-58d5-4bcc-aa42-00d95a6c7dd0 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:32.849439 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3a1f47-58d5-4bcc-aa42-00d95a6c7dd0 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:32.849439 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3a1f47-58d5-4bcc-aa42-00d95a6c7dd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:32.868513 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-7f3a1f47-58d5-4bcc-aa42-00d95a6c7dd0 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:32.869983 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-7f3a1f47-58d5-4bcc-aa42-00d95a6c7dd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:32.894572 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:32.939320 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:32.939826 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 376/763] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:32 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:12:32.941340 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3a1f47-58d5-4bcc-aa42-00d95a6c7dd0 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:32.954309 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3da0724-62b1-40fe-a867-20611e5c0bfc None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:33.000381 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f60e5a-2fe8-47bb-bced-856970a0ebbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:33.001458 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f60e5a-2fe8-47bb-bced-856970a0ebbc None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:33.001885 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f60e5a-2fe8-47bb-bced-856970a0ebbc None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:33.003149 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f60e5a-2fe8-47bb-bced-856970a0ebbc None admin] PATH_INFO: `/v3/services` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:33.004198 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f60e5a-2fe8-47bb-bced-856970a0ebbc None admin] RBAC: Adding query filter params (type=image) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:33.004647 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f60e5a-2fe8-47bb-bced-856970a0ebbc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:33.006373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f60e5a-2fe8-47bb-bced-856970a0ebbc None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:33.013239 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 388/764] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:12:32 2022] GET /identity/v3/services?type=image => generated 350 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:33.019256 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f60e5a-2fe8-47bb-bced-856970a0ebbc None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:33.063597 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3a1f47-58d5-4bcc-aa42-00d95a6c7dd0 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:33.111803 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e2cd91-cfb3-4eb9-9dab-c1aa8381aebd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:33.112691 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e2cd91-cfb3-4eb9-9dab-c1aa8381aebd None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:33.113145 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e2cd91-cfb3-4eb9-9dab-c1aa8381aebd None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:33.113563 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e2cd91-cfb3-4eb9-9dab-c1aa8381aebd None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:33.114857 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e2cd91-cfb3-4eb9-9dab-c1aa8381aebd None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:33.115455 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e2cd91-cfb3-4eb9-9dab-c1aa8381aebd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:33.117409 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e2cd91-cfb3-4eb9-9dab-c1aa8381aebd None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:33.124975 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 377/765] 10.222.0.42 () {56 vars in 1122 bytes} [Thu Oct 27 01:12:32 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:33.130212 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f60e5a-2fe8-47bb-bced-856970a0ebbc None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:33.133236 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e2cd91-cfb3-4eb9-9dab-c1aa8381aebd None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:33.166522 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:33.171213 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8004b05e-e7db-4af8-a6f6-a08f283226b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:33.172106 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8004b05e-e7db-4af8-a6f6-a08f283226b1 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:33.172701 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8004b05e-e7db-4af8-a6f6-a08f283226b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:33.173218 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8004b05e-e7db-4af8-a6f6-a08f283226b1 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:33.174569 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8004b05e-e7db-4af8-a6f6-a08f283226b1 None admin] RBAC: Adding query filter params (interface=public, service_id=a4fb9c1ad70e401eb96504258d8fbeaa, region_id=RegionOne) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:33.175115 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8004b05e-e7db-4af8-a6f6-a08f283226b1 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:33.177645 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8004b05e-e7db-4af8-a6f6-a08f283226b1 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:33.188277 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 389/766] 10.222.0.42 () {56 vars in 1251 bytes} [Thu Oct 27 01:12:33 2022] GET /identity/v3/endpoints?service_id=a4fb9c1ad70e401eb96504258d8fbeaa&interface=public®ion_id=RegionOne => generated 194 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:12:33.190430 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8004b05e-e7db-4af8-a6f6-a08f283226b1 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:33.216475 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:33.250158 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e2cd91-cfb3-4eb9-9dab-c1aa8381aebd None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:33.286572 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f642220e-9200-4fd7-a9a0-2e56bfee7343 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:33.287301 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f642220e-9200-4fd7-a9a0-2e56bfee7343 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:33.287499 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f642220e-9200-4fd7-a9a0-2e56bfee7343 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:33.287771 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f642220e-9200-4fd7-a9a0-2e56bfee7343 None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:33.288393 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f642220e-9200-4fd7-a9a0-2e56bfee7343 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:33.288691 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f642220e-9200-4fd7-a9a0-2e56bfee7343 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:33.290106 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f642220e-9200-4fd7-a9a0-2e56bfee7343 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:33.300022 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 378/767] 10.222.0.42 () {56 vars in 1122 bytes} [Thu Oct 27 01:12:33 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:33.300022 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f642220e-9200-4fd7-a9a0-2e56bfee7343 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:33.303529 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8004b05e-e7db-4af8-a6f6-a08f283226b1 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:33.340111 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec26f28c-f0f3-495f-9307-0d17ebd163eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:33.340724 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec26f28c-f0f3-495f-9307-0d17ebd163eb None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:33.341002 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec26f28c-f0f3-495f-9307-0d17ebd163eb None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:33.341357 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec26f28c-f0f3-495f-9307-0d17ebd163eb None admin] PATH_INFO: `/v3/services` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:33.341972 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec26f28c-f0f3-495f-9307-0d17ebd163eb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:33.343362 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec26f28c-f0f3-495f-9307-0d17ebd163eb None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:33.350287 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 390/768] 10.222.0.42 () {56 vars in 1087 bytes} [Thu Oct 27 01:12:33 2022] GET /identity/v3/services => generated 1957 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:12:33.350287 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec26f28c-f0f3-495f-9307-0d17ebd163eb None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:33.412199 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:33.428345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f642220e-9200-4fd7-a9a0-2e56bfee7343 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:33.467248 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a24001-1296-453c-8164-f6520cc67bbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:33.467883 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a24001-1296-453c-8164-f6520cc67bbf None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:33.468232 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a24001-1296-453c-8164-f6520cc67bbf None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:33.468602 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a24001-1296-453c-8164-f6520cc67bbf None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:33.469362 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a24001-1296-453c-8164-f6520cc67bbf None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:33.470884 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a24001-1296-453c-8164-f6520cc67bbf None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:33.487457 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.sql.core [None req-61a24001-1296-453c-8164-f6520cc67bbf None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 27 01:12:33.487457 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 27 01:12:33.487457 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [parameters: {'user_id': '668c6d33ff9c417f958ac660df539865', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 27 01:12:33.487457 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81587) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 27 01:12:33.488944 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:33.498039 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-61a24001-1296-453c-8164-f6520cc67bbf None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 27 01:12:33.498039 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:33.498039 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 27 01:12:33.498039 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 27 01:12:33.498039 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 27 01:12:33.498039 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 27 01:12:33.498039 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 27 01:12:33.498039 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application result = self._query(query) Oct 27 01:12:33.498039 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 27 01:12:33.498039 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application conn.query(q) Oct 27 01:12:33.498039 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 27 01:12:33.498039 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 27 01:12:33.498039 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 27 01:12:33.498039 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application result.read() Oct 27 01:12:33.498039 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 27 01:12:33.499162 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 27 01:12:33.499162 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 27 01:12:33.499162 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 27 01:12:33.499162 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 27 01:12:33.499162 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 27 01:12:33.499162 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 27 01:12:33.499162 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 27 01:12:33.499162 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 27 01:12:33.499162 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:33.499162 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 27 01:12:33.499162 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:33.499162 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:33.499162 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 27 01:12:33.499162 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 27 01:12:33.499162 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Oct 27 01:12:33.499162 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Oct 27 01:12:33.499162 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 27 01:12:33.499162 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application next(self.gen) Oct 27 01:12:33.500153 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1068, in _transaction_scope Oct 27 01:12:33.500153 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application yield resource Oct 27 01:12:33.500153 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 27 01:12:33.500153 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application next(self.gen) Oct 27 01:12:33.500153 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 670, in _session Oct 27 01:12:33.500153 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self.session.rollback() Oct 27 01:12:33.500153 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 27 01:12:33.500153 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self.force_reraise() Oct 27 01:12:33.500153 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 27 01:12:33.500153 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise self.value Oct 27 01:12:33.500153 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 667, in _session Oct 27 01:12:33.500153 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 27 01:12:33.500153 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 695, in _end_session_transaction Oct 27 01:12:33.500153 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application session.commit() Oct 27 01:12:33.500153 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 27 01:12:33.500153 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 27 01:12:33.500153 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 27 01:12:33.501036 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._prepare_impl() Oct 27 01:12:33.501036 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 27 01:12:33.501036 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self.session.flush() Oct 27 01:12:33.501036 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 27 01:12:33.501036 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._flush(objects) Oct 27 01:12:33.501036 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 27 01:12:33.501036 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 27 01:12:33.501036 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 27 01:12:33.501036 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application compat.raise_( Oct 27 01:12:33.501036 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 27 01:12:33.501036 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception Oct 27 01:12:33.501036 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 27 01:12:33.501036 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application flush_context.execute() Oct 27 01:12:33.501036 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 27 01:12:33.501036 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rec.execute(self) Oct 27 01:12:33.501036 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 27 01:12:33.501036 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 27 01:12:33.501947 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 27 01:12:33.501947 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 27 01:12:33.501947 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Oct 27 01:12:33.501947 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application result = connection._execute_20( Oct 27 01:12:33.501947 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 27 01:12:33.501947 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 27 01:12:33.501947 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 27 01:12:33.501947 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 27 01:12:33.501947 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 27 01:12:33.501947 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 27 01:12:33.501947 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 27 01:12:33.501947 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 27 01:12:33.501947 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 27 01:12:33.501947 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 27 01:12:33.501947 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 27 01:12:33.501947 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception Oct 27 01:12:33.502837 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 27 01:12:33.502837 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 27 01:12:33.502837 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 27 01:12:33.502837 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 27 01:12:33.502837 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 27 01:12:33.502837 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application result = self._query(query) Oct 27 01:12:33.502837 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 27 01:12:33.502837 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application conn.query(q) Oct 27 01:12:33.502837 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 27 01:12:33.502837 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 27 01:12:33.502837 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 27 01:12:33.502837 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application result.read() Oct 27 01:12:33.502837 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 27 01:12:33.502837 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 27 01:12:33.502837 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 27 01:12:33.502837 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 27 01:12:33.503709 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 27 01:12:33.503709 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 27 01:12:33.503709 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 27 01:12:33.503709 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 27 01:12:33.503709 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 27 01:12:33.503709 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 27 01:12:33.503709 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [parameters: {'user_id': '668c6d33ff9c417f958ac660df539865', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 27 01:12:33.503709 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 27 01:12:33.503709 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:33.503709 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 27 01:12:33.503709 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:33.503709 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:33.503709 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:33.503709 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:33.503709 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:33.504487 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:33.504487 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:33.504487 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:33.504487 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:33.504487 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:33.504487 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:33.504487 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:33.504487 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Oct 27 01:12:33.504487 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Oct 27 01:12:33.504487 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:33.504487 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:33.504487 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 27 01:12:33.504487 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:12:33.504487 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 27 01:12:33.504487 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:12:33.504487 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Oct 27 01:12:33.504487 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Oct 27 01:12:33.510713 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Oct 27 01:12:33.510713 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Oct 27 01:12:33.510713 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 27 01:12:33.510713 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 27 01:12:33.510713 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 27 01:12:33.510713 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:33.510713 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 379/769] 10.222.0.42 () {60 vars in 1131 bytes} [Thu Oct 27 01:12:33 2022] POST /identity/v3/users => generated 160 bytes in 205 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:33.510713 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a24001-1296-453c-8164-f6520cc67bbf None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:33.511175 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec26f28c-f0f3-495f-9307-0d17ebd163eb None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:33.546518 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8dcd889-3444-41c3-9fa4-fd8f1e5b4c2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:33.547129 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8dcd889-3444-41c3-9fa4-fd8f1e5b4c2b None admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:33.547406 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8dcd889-3444-41c3-9fa4-fd8f1e5b4c2b None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:33.547683 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8dcd889-3444-41c3-9fa4-fd8f1e5b4c2b None admin] PATH_INFO: `/v3/endpoints` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:33.548351 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8dcd889-3444-41c3-9fa4-fd8f1e5b4c2b None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:33.550315 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8dcd889-3444-41c3-9fa4-fd8f1e5b4c2b None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:33.567109 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 391/770] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:12:33 2022] POST /identity/v3/endpoints => generated 325 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:12:33.568266 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1175d2a-a407-4936-9eb4-56fbb304f2ef None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:33.568597 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1175d2a-a407-4936-9eb4-56fbb304f2ef None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:33.568881 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1175d2a-a407-4936-9eb4-56fbb304f2ef None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:33.574264 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 392/771] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:33.577150 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d745a56-0851-4766-a022-e0282e5c4552 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:33.577434 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d745a56-0851-4766-a022-e0282e5c4552 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:33.577704 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d745a56-0851-4766-a022-e0282e5c4552 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:33.598674 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-3d745a56-0851-4766-a022-e0282e5c4552 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:33.599535 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-3d745a56-0851-4766-a022-e0282e5c4552 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:33.645956 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a24001-1296-453c-8164-f6520cc67bbf None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:33.695969 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 393/772] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:33 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:12:33.707873 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e913fd-23ef-46f1-b7f2-03f2a3d291ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:33.708521 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e913fd-23ef-46f1-b7f2-03f2a3d291ad None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:33.708795 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e913fd-23ef-46f1-b7f2-03f2a3d291ad None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:33.709086 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e913fd-23ef-46f1-b7f2-03f2a3d291ad None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:33.718072 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e913fd-23ef-46f1-b7f2-03f2a3d291ad None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:33.719578 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e913fd-23ef-46f1-b7f2-03f2a3d291ad None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:33.733267 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-42e913fd-23ef-46f1-b7f2-03f2a3d291ad None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 27 01:12:33.733267 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:33.733267 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:33.733267 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:33.733267 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:33.733267 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:33.733267 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:33.733267 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:33.733267 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:33.733267 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:33.733267 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:33.733267 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:33.733267 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 27 01:12:33.733267 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 27 01:12:33.733267 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 27 01:12:33.733267 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 27 01:12:33.734299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:33.734299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:33.734299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 27 01:12:33.734299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:12:33.734299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 27 01:12:33.734299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:12:33.734299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:33.734299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:33.734299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:33.734299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:33.734299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:33.734299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:33.734299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:33.734299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:33.734299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:33.734299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:33.734299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:33.734299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:33.735937 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:33.735937 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:33.735937 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 27 01:12:33.735937 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 27 01:12:33.735937 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 27 01:12:33.735937 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 27 01:12:33.735937 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 27 01:12:33.735937 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 27 01:12:33.735937 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 27 01:12:33.735937 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:33.735937 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 380/773] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:12:33 2022] GET /identity/v3/users/cinder => generated 84 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 27 01:12:33.740812 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d745a56-0851-4766-a022-e0282e5c4552 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:33.825423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:33.856424 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d745a56-0851-4766-a022-e0282e5c4552 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:33.891990 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee5e02e-c489-4369-80ce-ccefcd07041a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:33.892648 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee5e02e-c489-4369-80ce-ccefcd07041a None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:33.892987 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee5e02e-c489-4369-80ce-ccefcd07041a None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:33.893291 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee5e02e-c489-4369-80ce-ccefcd07041a None admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:33.893990 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ee5e02e-c489-4369-80ce-ccefcd07041a None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:33.894289 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ee5e02e-c489-4369-80ce-ccefcd07041a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:33.895779 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ee5e02e-c489-4369-80ce-ccefcd07041a None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:33.932279 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 394/774] 10.222.0.42 () {56 vars in 1101 bytes} [Thu Oct 27 01:12:33 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:33.998058 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e913fd-23ef-46f1-b7f2-03f2a3d291ad None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:34.008746 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:34.117168 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e913fd-23ef-46f1-b7f2-03f2a3d291ad None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:34.155692 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3c7188-aa2a-40f6-a4be-9955453c648b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:34.157188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3c7188-aa2a-40f6-a4be-9955453c648b None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:34.157694 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3c7188-aa2a-40f6-a4be-9955453c648b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:34.158186 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3c7188-aa2a-40f6-a4be-9955453c648b None admin] PATH_INFO: `/v3/services/compute` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:34.159077 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3c7188-aa2a-40f6-a4be-9955453c648b None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:34.160840 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3c7188-aa2a-40f6-a4be-9955453c648b None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:34.170410 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-ee3c7188-aa2a-40f6-a4be-9955453c648b None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 27 01:12:34.170410 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:34.170410 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 27 01:12:34.170410 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 27 01:12:34.170410 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 27 01:12:34.170410 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 27 01:12:34.170410 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 27 01:12:34.170410 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:34.170410 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 27 01:12:34.170410 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:34.170410 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 27 01:12:34.170410 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:34.170410 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:34.170410 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:34.170410 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:34.170410 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:34.170410 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:34.171451 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:34.171451 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:34.171451 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:34.171451 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:34.171451 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:34.171451 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:34.171451 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 27 01:12:34.171451 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 27 01:12:34.171451 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 27 01:12:34.171451 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 27 01:12:34.171451 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:34.171451 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:34.171451 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:34.171451 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:34.171451 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:34.171451 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:34.172310 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:34.172310 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:34.172310 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:34.172310 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:34.172310 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:34.172310 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:34.172310 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:34.172310 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:34.172310 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:34.172310 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:34.172310 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 27 01:12:34.172310 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:34.172310 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 27 01:12:34.172310 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:34.173863 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 381/775] 10.222.0.42 () {56 vars in 1111 bytes} [Thu Oct 27 01:12:33 2022] GET /identity/v3/services/compute => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:34.180277 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee5e02e-c489-4369-80ce-ccefcd07041a None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:34.289242 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee5e02e-c489-4369-80ce-ccefcd07041a None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:34.318948 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:34.326210 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ae1cbd-7ae1-4af0-8a8f-36ec9db18394 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:34.326822 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ae1cbd-7ae1-4af0-8a8f-36ec9db18394 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:34.327102 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ae1cbd-7ae1-4af0-8a8f-36ec9db18394 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:34.327372 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ae1cbd-7ae1-4af0-8a8f-36ec9db18394 None admin] PATH_INFO: `/v3/services` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:34.327999 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ae1cbd-7ae1-4af0-8a8f-36ec9db18394 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:34.328299 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ae1cbd-7ae1-4af0-8a8f-36ec9db18394 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:34.329736 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ae1cbd-7ae1-4af0-8a8f-36ec9db18394 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:34.339879 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 395/776] 10.222.0.42 () {56 vars in 1112 bytes} [Thu Oct 27 01:12:34 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:34.341312 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3c7188-aa2a-40f6-a4be-9955453c648b None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:34.416504 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:34.447933 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3c7188-aa2a-40f6-a4be-9955453c648b None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:34.482278 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af9a11f-6222-4f53-b3af-9b3396ac812f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:34.482865 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af9a11f-6222-4f53-b3af-9b3396ac812f None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:34.483137 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af9a11f-6222-4f53-b3af-9b3396ac812f None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:34.483401 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af9a11f-6222-4f53-b3af-9b3396ac812f None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:34.484014 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af9a11f-6222-4f53-b3af-9b3396ac812f None admin] RBAC: Adding query filter params (type=compute) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:34.484307 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af9a11f-6222-4f53-b3af-9b3396ac812f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:34.485693 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af9a11f-6222-4f53-b3af-9b3396ac812f None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:34.491817 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 382/777] 10.222.0.42 () {56 vars in 1112 bytes} [Thu Oct 27 01:12:34 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:34.497282 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ae1cbd-7ae1-4af0-8a8f-36ec9db18394 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:34.598567 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ae1cbd-7ae1-4af0-8a8f-36ec9db18394 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:34.631771 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90270fd2-79fe-4ecd-8205-5aac7a120aec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:34.632365 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90270fd2-79fe-4ecd-8205-5aac7a120aec None admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:34.632639 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90270fd2-79fe-4ecd-8205-5aac7a120aec None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:34.632903 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90270fd2-79fe-4ecd-8205-5aac7a120aec None admin] PATH_INFO: `/v3/endpoints` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:34.633588 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90270fd2-79fe-4ecd-8205-5aac7a120aec None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:34.634948 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90270fd2-79fe-4ecd-8205-5aac7a120aec None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:34.661088 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 396/778] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:12:34 2022] POST /identity/v3/endpoints => generated 334 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:12:35.644036 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02c43cf-7d9e-48ee-b235-b612f94538bf None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:35.644568 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02c43cf-7d9e-48ee-b235-b612f94538bf None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:35.644568 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02c43cf-7d9e-48ee-b235-b612f94538bf None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:35.646068 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 383/779] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:35.651095 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d027fc-42d8-4769-8294-cccd536a0e26 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:35.651471 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d027fc-42d8-4769-8294-cccd536a0e26 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:35.651629 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d027fc-42d8-4769-8294-cccd536a0e26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:35.669758 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-c2d027fc-42d8-4769-8294-cccd536a0e26 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:35.670942 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-c2d027fc-42d8-4769-8294-cccd536a0e26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:35.720046 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:35.751270 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 397/780] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:35 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:12:35.833428 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44a61f5-79c8-4ff0-84ab-9c8722105284 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:35.833718 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44a61f5-79c8-4ff0-84ab-9c8722105284 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:35.833999 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44a61f5-79c8-4ff0-84ab-9c8722105284 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:35.835468 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 384/781] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:35.840501 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0acadbf9-1a61-40c8-8b54-a989f990d34d None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:35.840793 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0acadbf9-1a61-40c8-8b54-a989f990d34d None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:35.841088 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0acadbf9-1a61-40c8-8b54-a989f990d34d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:35.860697 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-0acadbf9-1a61-40c8-8b54-a989f990d34d None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:35.862423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-0acadbf9-1a61-40c8-8b54-a989f990d34d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:35.930404 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 398/782] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:35 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:12:36.026313 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44a61f5-79c8-4ff0-84ab-9c8722105284 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:36.131613 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44a61f5-79c8-4ff0-84ab-9c8722105284 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:36.165626 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8ebfaa-35c6-420d-bb6c-efcf4c96217a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:36.166535 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8ebfaa-35c6-420d-bb6c-efcf4c96217a None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:36.167016 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8ebfaa-35c6-420d-bb6c-efcf4c96217a None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:36.167489 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8ebfaa-35c6-420d-bb6c-efcf4c96217a None admin] PATH_INFO: `/v3/services/image` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:36.168276 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c8ebfaa-35c6-420d-bb6c-efcf4c96217a None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:36.169879 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c8ebfaa-35c6-420d-bb6c-efcf4c96217a None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:36.171810 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:36.180507 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-3c8ebfaa-35c6-420d-bb6c-efcf4c96217a None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 27 01:12:36.180507 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:36.180507 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 27 01:12:36.180507 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 27 01:12:36.180507 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 27 01:12:36.180507 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 27 01:12:36.180507 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 27 01:12:36.180507 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:36.180507 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 27 01:12:36.180507 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:36.180507 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 27 01:12:36.180507 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:36.180507 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:36.180507 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:36.180507 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:36.180507 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:36.180507 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:36.181764 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:36.181764 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:36.181764 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:36.181764 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:36.181764 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:36.181764 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:36.181764 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 27 01:12:36.181764 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 27 01:12:36.181764 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 27 01:12:36.181764 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 27 01:12:36.181764 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:36.181764 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:36.181764 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:36.181764 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:36.181764 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:36.181764 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:36.186148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:36.186148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:36.186148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:36.186148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:36.186148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:36.186148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:36.186148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:36.186148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:36.186148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:36.186148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:36.186148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 27 01:12:36.186148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:12:36.186148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 27 01:12:36.186148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:36.186148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 385/783] 10.222.0.42 () {56 vars in 1105 bytes} [Thu Oct 27 01:12:36 2022] GET /identity/v3/services/image => generated 86 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:36.188066 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0acadbf9-1a61-40c8-8b54-a989f990d34d None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:36.200339 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8ebfaa-35c6-420d-bb6c-efcf4c96217a None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:36.299703 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0acadbf9-1a61-40c8-8b54-a989f990d34d None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:36.305722 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8ebfaa-35c6-420d-bb6c-efcf4c96217a None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:36.309106 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:36.339918 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd3de93-65c3-44a8-887b-de7d66ab1e17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:36.340568 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd3de93-65c3-44a8-887b-de7d66ab1e17 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:36.340855 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd3de93-65c3-44a8-887b-de7d66ab1e17 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:36.341157 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd3de93-65c3-44a8-887b-de7d66ab1e17 None admin] PATH_INFO: `/v3/services` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:36.341973 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd3de93-65c3-44a8-887b-de7d66ab1e17 None admin] RBAC: Adding query filter params (name=image) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:36.342278 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd3de93-65c3-44a8-887b-de7d66ab1e17 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:36.343665 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd3de93-65c3-44a8-887b-de7d66ab1e17 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:36.343959 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d6e8c3-edcc-4e33-bcdd-763ee149db6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:36.344588 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d6e8c3-edcc-4e33-bcdd-763ee149db6a None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:36.344863 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d6e8c3-edcc-4e33-bcdd-763ee149db6a None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:36.345161 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d6e8c3-edcc-4e33-bcdd-763ee149db6a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:36.346320 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64d6e8c3-edcc-4e33-bcdd-763ee149db6a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:36.347726 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64d6e8c3-edcc-4e33-bcdd-763ee149db6a None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:36.349326 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 386/784] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:12:36 2022] GET /identity/v3/domains/default => generated 199 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:12:36.349926 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 399/785] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:12:36 2022] GET /identity/v3/services?name=image => generated 122 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:36.355024 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d6e8c3-edcc-4e33-bcdd-763ee149db6a None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:36.355641 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd3de93-65c3-44a8-887b-de7d66ab1e17 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:36.429846 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:36.465423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd3de93-65c3-44a8-887b-de7d66ab1e17 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:36.466756 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d6e8c3-edcc-4e33-bcdd-763ee149db6a None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:36.501244 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe80cb4-1ca4-475b-948e-e62a0d49495b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:36.501968 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe80cb4-1ca4-475b-948e-e62a0d49495b None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:36.502370 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe80cb4-1ca4-475b-948e-e62a0d49495b None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:36.502513 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59866cef-51f2-4998-b731-fd335755f5d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:36.502767 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe80cb4-1ca4-475b-948e-e62a0d49495b None admin] PATH_INFO: `/v3/services` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:36.503193 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59866cef-51f2-4998-b731-fd335755f5d5 None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:36.503471 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59866cef-51f2-4998-b731-fd335755f5d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:36.503566 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe80cb4-1ca4-475b-948e-e62a0d49495b None admin] RBAC: Adding query filter params (type=image) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:36.503742 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59866cef-51f2-4998-b731-fd335755f5d5 None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:36.503949 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe80cb4-1ca4-475b-948e-e62a0d49495b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:36.504386 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59866cef-51f2-4998-b731-fd335755f5d5 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:36.505476 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe80cb4-1ca4-475b-948e-e62a0d49495b None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:36.505818 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59866cef-51f2-4998-b731-fd335755f5d5 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:36.510847 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 400/786] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:12:36 2022] GET /identity/v3/services?type=image => generated 350 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:12:36.516393 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe80cb4-1ca4-475b-948e-e62a0d49495b None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:36.541353 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 387/787] 10.222.0.42 () {60 vars in 1141 bytes} [Thu Oct 27 01:12:36 2022] POST /identity/v3/projects => generated 311 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:12:36.554205 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:36.629077 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe80cb4-1ca4-475b-948e-e62a0d49495b None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:36.665955 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02fde29-14da-41ad-9f0a-152cf5db7881 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:36.666672 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02fde29-14da-41ad-9f0a-152cf5db7881 None admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:36.666932 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02fde29-14da-41ad-9f0a-152cf5db7881 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:36.667124 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02fde29-14da-41ad-9f0a-152cf5db7881 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:36.667804 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c02fde29-14da-41ad-9f0a-152cf5db7881 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:36.669216 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c02fde29-14da-41ad-9f0a-152cf5db7881 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:36.694891 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 401/788] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:12:36 2022] POST /identity/v3/endpoints => generated 327 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:12:36.955763 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da6737a-fd12-41da-b530-33e00c0b11ba None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:36.956366 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da6737a-fd12-41da-b530-33e00c0b11ba None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:36.956366 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da6737a-fd12-41da-b530-33e00c0b11ba None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:36.957827 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 388/789] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 27 01:12:36.962861 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d757367-52ca-4247-a6a7-d139c4bd2598 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:36.963138 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d757367-52ca-4247-a6a7-d139c4bd2598 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:36.963419 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d757367-52ca-4247-a6a7-d139c4bd2598 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:36.982472 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-3d757367-52ca-4247-a6a7-d139c4bd2598 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:36.983647 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-3d757367-52ca-4247-a6a7-d139c4bd2598 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:37.058623 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 402/790] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:36 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:12:37.326644 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da6737a-fd12-41da-b530-33e00c0b11ba None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:37.430079 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da6737a-fd12-41da-b530-33e00c0b11ba None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:37.447883 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:37.467730 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fea72fb-a7e8-4b86-8bd3-5ac2dadbb3bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:37.468413 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fea72fb-a7e8-4b86-8bd3-5ac2dadbb3bb None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:37.468695 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fea72fb-a7e8-4b86-8bd3-5ac2dadbb3bb None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:37.468963 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fea72fb-a7e8-4b86-8bd3-5ac2dadbb3bb None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:37.479929 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fea72fb-a7e8-4b86-8bd3-5ac2dadbb3bb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:12:37.484677 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-5fea72fb-a7e8-4b86-8bd3-5ac2dadbb3bb None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 27 01:12:37.484677 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:37.484677 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:37.484677 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:37.484677 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:37.484677 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:37.484677 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:37.484677 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:37.484677 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:37.484677 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:37.484677 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:37.484677 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:37.484677 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:12:37.484677 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:12:37.484677 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:12:37.484677 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:12:37.489228 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:12:37.489228 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:12:37.489228 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:12:37.489228 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:12:37.489228 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:37.489228 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:37.489228 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:37.489228 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:37.489228 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:37.489228 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:37.489228 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:37.489228 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:37.489228 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:37.489228 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:37.489228 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:37.489228 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:37.490180 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:37.490180 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:37.490180 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:37.490180 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:37.490180 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:12:37.490180 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:12:37.490180 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:12:37.490180 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:12:37.490180 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:12:37.490180 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:12:37.490180 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 27 01:12:37.490180 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:37.490180 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 389/791] 10.222.0.42 () {56 vars in 1120 bytes} [Thu Oct 27 01:12:37 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:37.492020 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d757367-52ca-4247-a6a7-d139c4bd2598 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:37.511664 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:37.597729 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d757367-52ca-4247-a6a7-d139c4bd2598 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:37.633814 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7590ac6f-e373-4aa2-bc73-d28608d0e3db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:37.634419 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7590ac6f-e373-4aa2-bc73-d28608d0e3db None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:37.634695 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7590ac6f-e373-4aa2-bc73-d28608d0e3db None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:37.634962 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7590ac6f-e373-4aa2-bc73-d28608d0e3db None admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:37.635614 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7590ac6f-e373-4aa2-bc73-d28608d0e3db None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:37.635913 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7590ac6f-e373-4aa2-bc73-d28608d0e3db None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:37.637312 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7590ac6f-e373-4aa2-bc73-d28608d0e3db None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:37.645283 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 403/792] 10.222.0.42 () {56 vars in 1115 bytes} [Thu Oct 27 01:12:37 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:37.650875 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fea72fb-a7e8-4b86-8bd3-5ac2dadbb3bb None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:37.749498 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fea72fb-a7e8-4b86-8bd3-5ac2dadbb3bb None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:37.783805 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c692009d-b32d-447e-81d8-1b046f41741f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:37.784505 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c692009d-b32d-447e-81d8-1b046f41741f None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:37.784780 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c692009d-b32d-447e-81d8-1b046f41741f None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:37.785050 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c692009d-b32d-447e-81d8-1b046f41741f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:37.794855 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c692009d-b32d-447e-81d8-1b046f41741f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:37.796290 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c692009d-b32d-447e-81d8-1b046f41741f None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:37.819452 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-c692009d-b32d-447e-81d8-1b046f41741f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:12:37.819452 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:37.819452 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:37.819452 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:37.819452 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:37.819452 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:37.819452 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:37.819452 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:37.819452 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:37.819452 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:37.819452 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:37.819452 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:37.819452 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:12:37.819452 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:12:37.819452 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:12:37.819452 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:12:37.820367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:37.820367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:37.820367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:37.820367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:37.820367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:37.820367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:37.820367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:37.820367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:37.820367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:37.820367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:37.820367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:37.820367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:37.820367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:37.820367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:37.820367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:37.820367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:37.820367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:12:37.821304 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:12:37.821304 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:12:37.821304 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:37.821304 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 390/793] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:12:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:37.827265 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7590ac6f-e373-4aa2-bc73-d28608d0e3db None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:37.908699 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:37.933034 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7590ac6f-e373-4aa2-bc73-d28608d0e3db None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:37.966576 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b55906-1b51-4fa4-8b8a-aea844d1490f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:37.967173 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b55906-1b51-4fa4-8b8a-aea844d1490f None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:37.967444 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b55906-1b51-4fa4-8b8a-aea844d1490f None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:37.967709 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b55906-1b51-4fa4-8b8a-aea844d1490f None admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:37.968335 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b55906-1b51-4fa4-8b8a-aea844d1490f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:37.968627 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b55906-1b51-4fa4-8b8a-aea844d1490f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:37.970003 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b55906-1b51-4fa4-8b8a-aea844d1490f None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:37.989123 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 404/794] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:12:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:37.990948 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c692009d-b32d-447e-81d8-1b046f41741f None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:38.022676 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:38.093231 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c692009d-b32d-447e-81d8-1b046f41741f None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:38.126638 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4856504e-d16b-4705-a4cb-5dc0f517ec8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:38.128392 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4856504e-d16b-4705-a4cb-5dc0f517ec8b None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:38.128668 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4856504e-d16b-4705-a4cb-5dc0f517ec8b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:38.128934 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4856504e-d16b-4705-a4cb-5dc0f517ec8b None admin] PATH_INFO: `/v3/users/nova` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:38.136104 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4856504e-d16b-4705-a4cb-5dc0f517ec8b None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:38.137569 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4856504e-d16b-4705-a4cb-5dc0f517ec8b None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:38.148820 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-4856504e-d16b-4705-a4cb-5dc0f517ec8b None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 27 01:12:38.148820 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:38.148820 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:38.148820 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:38.148820 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:38.148820 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:38.148820 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:38.148820 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:38.148820 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:38.148820 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:38.148820 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:38.148820 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:38.148820 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 27 01:12:38.148820 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 27 01:12:38.148820 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 27 01:12:38.148820 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 27 01:12:38.149899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:38.149899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:38.149899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 27 01:12:38.149899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:12:38.149899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 27 01:12:38.149899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:12:38.149899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:38.149899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:38.149899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:38.149899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:38.149899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:38.149899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:38.149899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:38.149899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:38.149899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:38.149899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:38.149899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:38.149899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:38.150924 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:38.150924 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:38.150924 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 27 01:12:38.150924 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 27 01:12:38.150924 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 27 01:12:38.150924 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 27 01:12:38.150924 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 27 01:12:38.150924 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 27 01:12:38.150924 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 27 01:12:38.150924 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:38.150924 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 391/795] 10.222.0.42 () {56 vars in 1093 bytes} [Thu Oct 27 01:12:37 2022] GET /identity/v3/users/nova => generated 82 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 27 01:12:38.157358 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b55906-1b51-4fa4-8b8a-aea844d1490f None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:38.259934 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b55906-1b51-4fa4-8b8a-aea844d1490f None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:38.294001 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323d4f3b-035a-447e-b9f7-778878048eb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:38.294609 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323d4f3b-035a-447e-b9f7-778878048eb4 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:38.294882 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323d4f3b-035a-447e-b9f7-778878048eb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:38.295152 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323d4f3b-035a-447e-b9f7-778878048eb4 None admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:38.295866 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323d4f3b-035a-447e-b9f7-778878048eb4 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:38.296168 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323d4f3b-035a-447e-b9f7-778878048eb4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:38.297629 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323d4f3b-035a-447e-b9f7-778878048eb4 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:38.305700 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:38.317582 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 405/796] 10.222.0.42 () {56 vars in 1133 bytes} [Thu Oct 27 01:12:38 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:38.323169 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4856504e-d16b-4705-a4cb-5dc0f517ec8b None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:38.414880 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:38.426178 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4856504e-d16b-4705-a4cb-5dc0f517ec8b None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:38.459448 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d03de26-69d2-4741-8b27-635164c4b856 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:38.460073 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d03de26-69d2-4741-8b27-635164c4b856 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:38.460347 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d03de26-69d2-4741-8b27-635164c4b856 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:38.460612 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d03de26-69d2-4741-8b27-635164c4b856 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:38.471711 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d03de26-69d2-4741-8b27-635164c4b856 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:38.473368 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d03de26-69d2-4741-8b27-635164c4b856 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:38.488083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-1d03de26-69d2-4741-8b27-635164c4b856 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:12:38.488083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:38.488083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:38.488083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:38.488083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:38.488083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:38.488083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:38.488083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:38.488083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:38.488083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:38.488083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:38.488083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:38.488083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:12:38.488083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:12:38.488083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:12:38.488083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:12:38.489071 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:38.489071 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:38.489071 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:38.489071 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:38.489071 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:38.489071 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:38.489071 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:38.489071 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:38.489071 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:38.489071 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:38.489071 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:38.489071 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:38.489071 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:38.489071 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:38.489071 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:38.489071 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:38.489071 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:12:38.490014 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:12:38.490014 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:12:38.490014 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:38.490924 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 392/797] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:12:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:38.494914 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323d4f3b-035a-447e-b9f7-778878048eb4 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:38.599134 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323d4f3b-035a-447e-b9f7-778878048eb4 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:38.633297 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9210835-cd47-4e2a-aa9a-92f7c0245ef6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:38.633909 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9210835-cd47-4e2a-aa9a-92f7c0245ef6 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:38.634188 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9210835-cd47-4e2a-aa9a-92f7c0245ef6 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:38.634460 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9210835-cd47-4e2a-aa9a-92f7c0245ef6 None admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:38.635095 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9210835-cd47-4e2a-aa9a-92f7c0245ef6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:38.635398 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9210835-cd47-4e2a-aa9a-92f7c0245ef6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:38.636766 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9210835-cd47-4e2a-aa9a-92f7c0245ef6 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:38.653164 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 406/798] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:12:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:38.654719 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d03de26-69d2-4741-8b27-635164c4b856 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:38.761419 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d03de26-69d2-4741-8b27-635164c4b856 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:38.800848 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a81897-2dd8-4e1c-8e82-1bbd54c88ffc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:38.800848 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a81897-2dd8-4e1c-8e82-1bbd54c88ffc None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:38.800848 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a81897-2dd8-4e1c-8e82-1bbd54c88ffc None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:38.800848 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a81897-2dd8-4e1c-8e82-1bbd54c88ffc None admin] PATH_INFO: `/v3/projects/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:38.805387 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48094bc6-acc3-4dee-92fe-2f3cf5bb2c4a None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:38.805669 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48094bc6-acc3-4dee-92fe-2f3cf5bb2c4a None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:38.805944 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48094bc6-acc3-4dee-92fe-2f3cf5bb2c4a None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:38.807770 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 407/799] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 27 01:12:38.812804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a81897-2dd8-4e1c-8e82-1bbd54c88ffc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:38.813368 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca20d55-4617-4e88-89ae-f86edce6bd09 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:38.813653 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca20d55-4617-4e88-89ae-f86edce6bd09 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:38.813928 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca20d55-4617-4e88-89ae-f86edce6bd09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:38.815909 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a81897-2dd8-4e1c-8e82-1bbd54c88ffc None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:38.828680 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-56a81897-2dd8-4e1c-8e82-1bbd54c88ffc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:12:38.828680 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:38.828680 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:38.828680 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:38.828680 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:38.828680 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:38.828680 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:38.828680 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:38.828680 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:38.828680 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:38.828680 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:38.828680 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:38.828680 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 27 01:12:38.828680 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 27 01:12:38.828680 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 27 01:12:38.829778 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 27 01:12:38.829778 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:38.829778 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:38.829778 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:38.829778 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:38.829778 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:38.829778 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:38.829778 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:38.829778 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:38.829778 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:38.829778 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:38.829778 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:38.829778 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:38.829778 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:38.829778 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:38.829778 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:38.829778 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:38.830718 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 27 01:12:38.830718 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 27 01:12:38.830718 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 27 01:12:38.830718 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 27 01:12:38.830718 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 27 01:12:38.830718 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 27 01:12:38.830718 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:12:38.830718 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:38.836299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 393/800] 10.222.0.42 () {56 vars in 1111 bytes} [Thu Oct 27 01:12:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:38.836651 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a81897-2dd8-4e1c-8e82-1bbd54c88ffc None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:38.836759 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-6ca20d55-4617-4e88-89ae-f86edce6bd09 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:38.837990 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-6ca20d55-4617-4e88-89ae-f86edce6bd09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:38.839566 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:38.893456 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:38.911478 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 408/801] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:38 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 27 01:12:38.912730 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e849bd12-9c55-436c-b47f-00c3c379b796 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:38.913013 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e849bd12-9c55-436c-b47f-00c3c379b796 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:38.913317 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e849bd12-9c55-436c-b47f-00c3c379b796 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:38.914560 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 409/802] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:38.919833 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1eb691c-2268-4390-bb64-b77406623927 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:38.920278 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1eb691c-2268-4390-bb64-b77406623927 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:38.920703 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1eb691c-2268-4390-bb64-b77406623927 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:38.940483 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-b1eb691c-2268-4390-bb64-b77406623927 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:38.942139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-b1eb691c-2268-4390-bb64-b77406623927 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:38.944356 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a81897-2dd8-4e1c-8e82-1bbd54c88ffc None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:38.978532 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f484ed-130b-4730-900a-c5ca95dca53b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:38.979135 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f484ed-130b-4730-900a-c5ca95dca53b None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:38.979420 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f484ed-130b-4730-900a-c5ca95dca53b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:38.979682 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f484ed-130b-4730-900a-c5ca95dca53b None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:38.980368 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6f484ed-130b-4730-900a-c5ca95dca53b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:38.980664 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6f484ed-130b-4730-900a-c5ca95dca53b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:38.982125 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6f484ed-130b-4730-900a-c5ca95dca53b None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:38.994122 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 394/803] 10.222.0.42 () {56 vars in 1148 bytes} [Thu Oct 27 01:12:38 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:38.999916 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f484ed-130b-4730-900a-c5ca95dca53b None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:39.015370 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 410/804] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:38 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 27 01:12:39.063782 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:39.103078 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f484ed-130b-4730-900a-c5ca95dca53b None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:39.136277 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e95322f-15d3-43dd-85dd-248f0f42f004 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:39.136925 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e95322f-15d3-43dd-85dd-248f0f42f004 None admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:39.137253 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e95322f-15d3-43dd-85dd-248f0f42f004 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:39.137525 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e95322f-15d3-43dd-85dd-248f0f42f004 None admin] PATH_INFO: `/v3/projects/23aad848d47c4d27ad4c138ff869e40f/users/51033d56f85f4282a49ded6338440677/roles/04d145c26220455b969c13d2d24e84e8` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:39.139650 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e95322f-15d3-43dd-85dd-248f0f42f004 None admin] RBAC: Authorizing `identity:create_grant(project_id=23aad848d47c4d27ad4c138ff869e40f, user_id=51033d56f85f4282a49ded6338440677, role_id=04d145c26220455b969c13d2d24e84e8)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:39.141080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e95322f-15d3-43dd-85dd-248f0f42f004 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:39.153774 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 395/805] 10.222.0.42 () {58 vars in 1439 bytes} [Thu Oct 27 01:12:38 2022] PUT /identity/v3/projects/23aad848d47c4d27ad4c138ff869e40f/users/51033d56f85f4282a49ded6338440677/roles/04d145c26220455b969c13d2d24e84e8 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:12:39.185763 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1eb691c-2268-4390-bb64-b77406623927 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:39.289636 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e95322f-15d3-43dd-85dd-248f0f42f004 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:39.295626 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1eb691c-2268-4390-bb64-b77406623927 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:39.319225 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:39.343052 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4d44a4-b3d6-453c-bfa8-0689d1bf19da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:39.343679 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4d44a4-b3d6-453c-bfa8-0689d1bf19da None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:39.343955 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4d44a4-b3d6-453c-bfa8-0689d1bf19da None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:39.344226 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4d44a4-b3d6-453c-bfa8-0689d1bf19da None admin] PATH_INFO: `/v3/domains/default` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:39.345350 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be4d44a4-b3d6-453c-bfa8-0689d1bf19da None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:39.346852 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be4d44a4-b3d6-453c-bfa8-0689d1bf19da None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:39.348451 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 411/806] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:12:39 2022] GET /identity/v3/domains/default => generated 199 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:12:39.354339 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4d44a4-b3d6-453c-bfa8-0689d1bf19da None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:39.410360 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e95322f-15d3-43dd-85dd-248f0f42f004 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:39.435535 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:39.453386 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d6e7ab-8088-4733-8f66-7ed19254c8f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:39.453993 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d6e7ab-8088-4733-8f66-7ed19254c8f5 None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:39.454272 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d6e7ab-8088-4733-8f66-7ed19254c8f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:39.454542 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d6e7ab-8088-4733-8f66-7ed19254c8f5 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:39.455754 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d6e7ab-8088-4733-8f66-7ed19254c8f5 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:39.457143 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d6e7ab-8088-4733-8f66-7ed19254c8f5 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:39.459446 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4d44a4-b3d6-453c-bfa8-0689d1bf19da None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:39.466153 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.sql.core [None req-c0d6e7ab-8088-4733-8f66-7ed19254c8f5 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 27 01:12:39.466153 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 27 01:12:39.466153 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [parameters: {'id': 'd695c03270de413fb8b96e8e00e690e3', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 27 01:12:39.466153 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81587) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 27 01:12:39.475160 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-c0d6e7ab-8088-4733-8f66-7ed19254c8f5 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 27 01:12:39.475160 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:39.475160 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 27 01:12:39.475160 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 27 01:12:39.475160 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 27 01:12:39.475160 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 27 01:12:39.475160 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 27 01:12:39.475160 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application result = self._query(query) Oct 27 01:12:39.475160 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 27 01:12:39.475160 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application conn.query(q) Oct 27 01:12:39.475160 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 27 01:12:39.475160 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 27 01:12:39.475160 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 27 01:12:39.475160 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application result.read() Oct 27 01:12:39.476082 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 27 01:12:39.476082 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 27 01:12:39.476082 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 27 01:12:39.476082 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 27 01:12:39.476082 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 27 01:12:39.476082 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 27 01:12:39.476082 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 27 01:12:39.476082 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 27 01:12:39.476082 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 27 01:12:39.476082 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:39.476082 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 27 01:12:39.476082 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:39.476082 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:39.476082 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 27 01:12:39.476082 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 27 01:12:39.476082 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 27 01:12:39.476082 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 27 01:12:39.476082 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 27 01:12:39.481845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application next(self.gen) Oct 27 01:12:39.481845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1068, in _transaction_scope Oct 27 01:12:39.481845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application yield resource Oct 27 01:12:39.481845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 27 01:12:39.481845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application next(self.gen) Oct 27 01:12:39.481845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 670, in _session Oct 27 01:12:39.481845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self.session.rollback() Oct 27 01:12:39.481845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 27 01:12:39.481845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self.force_reraise() Oct 27 01:12:39.481845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 27 01:12:39.481845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise self.value Oct 27 01:12:39.481845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 667, in _session Oct 27 01:12:39.481845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 27 01:12:39.481845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 695, in _end_session_transaction Oct 27 01:12:39.481845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application session.commit() Oct 27 01:12:39.481845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 27 01:12:39.481845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 27 01:12:39.482899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 27 01:12:39.482899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._prepare_impl() Oct 27 01:12:39.482899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 27 01:12:39.482899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self.session.flush() Oct 27 01:12:39.482899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 27 01:12:39.482899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._flush(objects) Oct 27 01:12:39.482899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 27 01:12:39.482899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 27 01:12:39.482899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 27 01:12:39.482899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application compat.raise_( Oct 27 01:12:39.482899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 27 01:12:39.482899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception Oct 27 01:12:39.482899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 27 01:12:39.482899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application flush_context.execute() Oct 27 01:12:39.482899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 27 01:12:39.482899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rec.execute(self) Oct 27 01:12:39.482899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 27 01:12:39.484025 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 27 01:12:39.484025 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 27 01:12:39.484025 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 27 01:12:39.484025 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 27 01:12:39.484025 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 27 01:12:39.484025 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 27 01:12:39.484025 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 27 01:12:39.484025 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 27 01:12:39.484025 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 27 01:12:39.484025 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 27 01:12:39.484025 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 27 01:12:39.484025 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 27 01:12:39.484025 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 27 01:12:39.484025 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 27 01:12:39.484025 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 27 01:12:39.484999 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 27 01:12:39.484999 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception Oct 27 01:12:39.484999 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 27 01:12:39.484999 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 27 01:12:39.484999 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 27 01:12:39.484999 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 27 01:12:39.484999 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 27 01:12:39.484999 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application result = self._query(query) Oct 27 01:12:39.484999 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 27 01:12:39.484999 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application conn.query(q) Oct 27 01:12:39.484999 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 27 01:12:39.484999 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 27 01:12:39.484999 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 27 01:12:39.484999 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application result.read() Oct 27 01:12:39.484999 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 27 01:12:39.484999 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 27 01:12:39.484999 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 27 01:12:39.486144 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 27 01:12:39.486144 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 27 01:12:39.486144 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 27 01:12:39.486144 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 27 01:12:39.486144 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 27 01:12:39.486144 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 27 01:12:39.486144 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 27 01:12:39.486144 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [parameters: {'id': 'd695c03270de413fb8b96e8e00e690e3', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 27 01:12:39.486144 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 27 01:12:39.486144 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:39.486144 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 27 01:12:39.486144 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:39.486144 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:39.486144 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 27 01:12:39.486144 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 27 01:12:39.487646 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 27 01:12:39.487646 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 27 01:12:39.487646 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name Default. Oct 27 01:12:39.487646 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:39.487646 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 27 01:12:39.487646 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:39.487646 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:39.487646 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:39.487646 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:39.487646 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:39.487646 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:39.487646 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:39.487646 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:39.487646 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:39.487646 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:39.487646 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:39.487646 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:39.488736 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 128, in post Oct 27 01:12:39.488736 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_domain( Oct 27 01:12:39.488736 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:39.488736 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:39.488736 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 765, in create_domain Oct 27 01:12:39.488736 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application is_domain_project = self.create_project( Oct 27 01:12:39.488736 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:39.488736 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:39.488736 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 27 01:12:39.488736 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 27 01:12:39.488736 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 27 01:12:39.488736 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:39.488736 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 396/807] 10.222.0.42 () {60 vars in 1137 bytes} [Thu Oct 27 01:12:39 2022] POST /identity/v3/domains => generated 190 bytes in 188 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Oct 27 01:12:39.490624 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d6e7ab-8088-4733-8f66-7ed19254c8f5 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:39.499671 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba28983-ad5f-47ef-8d05-22c0f4217bcf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:39.500281 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba28983-ad5f-47ef-8d05-22c0f4217bcf None admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:39.500557 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba28983-ad5f-47ef-8d05-22c0f4217bcf None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:39.500825 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba28983-ad5f-47ef-8d05-22c0f4217bcf None admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:39.501511 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba28983-ad5f-47ef-8d05-22c0f4217bcf None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:39.503179 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba28983-ad5f-47ef-8d05-22c0f4217bcf None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:39.534719 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 412/808] 10.222.0.42 () {60 vars in 1132 bytes} [Thu Oct 27 01:12:39 2022] POST /identity/v3/users => generated 285 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:12:39.602773 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d6e7ab-8088-4733-8f66-7ed19254c8f5 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:39.640205 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44fc238b-e8bc-407a-be6e-974200154992 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:39.640856 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fc238b-e8bc-407a-be6e-974200154992 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:39.641209 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fc238b-e8bc-407a-be6e-974200154992 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:39.641458 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fc238b-e8bc-407a-be6e-974200154992 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:39.653337 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44fc238b-e8bc-407a-be6e-974200154992 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:39.654867 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44fc238b-e8bc-407a-be6e-974200154992 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:39.671801 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-44fc238b-e8bc-407a-be6e-974200154992 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:12:39.671801 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:39.671801 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:39.671801 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:39.671801 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:39.671801 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:39.671801 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:39.671801 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:39.671801 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:39.671801 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:39.671801 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:39.671801 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:39.671801 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:12:39.671801 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:12:39.671801 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:12:39.671801 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:12:39.672804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:39.672804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:39.672804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:39.672804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:39.672804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:39.672804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:39.672804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:39.672804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:39.672804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:39.672804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:39.672804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:39.672804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:39.672804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:39.672804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:39.672804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:39.672804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:39.672804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:12:39.678984 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:12:39.678984 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:12:39.678984 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:39.678984 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 397/809] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:12:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:39.679320 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba28983-ad5f-47ef-8d05-22c0f4217bcf None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:39.725209 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:39.786587 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba28983-ad5f-47ef-8d05-22c0f4217bcf None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:39.826882 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec45daa3-5fc7-46a0-a3fd-4595fc9e1a26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:39.827495 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec45daa3-5fc7-46a0-a3fd-4595fc9e1a26 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:39.827769 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec45daa3-5fc7-46a0-a3fd-4595fc9e1a26 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:39.828037 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec45daa3-5fc7-46a0-a3fd-4595fc9e1a26 None admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:39.828676 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec45daa3-5fc7-46a0-a3fd-4595fc9e1a26 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:39.828976 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec45daa3-5fc7-46a0-a3fd-4595fc9e1a26 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:39.830585 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec45daa3-5fc7-46a0-a3fd-4595fc9e1a26 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:39.842595 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 413/810] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:12:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:41.347896 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06477111-665c-46db-b424-e521144ddfb2 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:41.348484 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06477111-665c-46db-b424-e521144ddfb2 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:41.348484 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06477111-665c-46db-b424-e521144ddfb2 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:41.350030 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 398/811] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:41.354918 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86aa2c2c-959c-4eab-9e5d-e41b493a4cc0 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:41.355328 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86aa2c2c-959c-4eab-9e5d-e41b493a4cc0 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:41.355464 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86aa2c2c-959c-4eab-9e5d-e41b493a4cc0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:41.374454 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-86aa2c2c-959c-4eab-9e5d-e41b493a4cc0 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:41.375629 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-86aa2c2c-959c-4eab-9e5d-e41b493a4cc0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:41.441153 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 414/812] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:41 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:12:41.712065 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06477111-665c-46db-b424-e521144ddfb2 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:41.750515 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f6db78-e7a7-4e62-acd4-fe0654576fb4 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:41.750796 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f6db78-e7a7-4e62-acd4-fe0654576fb4 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:41.751069 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f6db78-e7a7-4e62-acd4-fe0654576fb4 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:41.752534 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 415/813] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:41.753832 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:41.758848 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d83851e-314d-47b3-9557-97fb2bdd3754 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:41.759135 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d83851e-314d-47b3-9557-97fb2bdd3754 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:41.759409 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d83851e-314d-47b3-9557-97fb2bdd3754 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:41.777478 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-3d83851e-314d-47b3-9557-97fb2bdd3754 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:41.778665 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-3d83851e-314d-47b3-9557-97fb2bdd3754 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:41.804415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:41.820168 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06477111-665c-46db-b424-e521144ddfb2 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:41.848940 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 416/814] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:41 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 27 01:12:41.856199 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a16646a-c4fe-47d9-b54e-0040e2040c76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:41.856815 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a16646a-c4fe-47d9-b54e-0040e2040c76 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:41.857112 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a16646a-c4fe-47d9-b54e-0040e2040c76 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:41.857381 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a16646a-c4fe-47d9-b54e-0040e2040c76 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:41.867695 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a16646a-c4fe-47d9-b54e-0040e2040c76 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:12:41.872447 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-7a16646a-c4fe-47d9-b54e-0040e2040c76 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 27 01:12:41.872447 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:41.872447 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:41.872447 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:41.872447 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:41.872447 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:41.872447 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:41.872447 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:41.872447 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:41.872447 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:41.872447 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:41.872447 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:41.872447 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:12:41.872447 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:12:41.872447 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:12:41.872447 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:12:41.877643 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:12:41.877643 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:12:41.877643 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:12:41.877643 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:12:41.877643 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:41.877643 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:41.877643 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:41.877643 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:41.877643 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:41.877643 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:41.877643 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:41.877643 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:41.877643 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:41.877643 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:41.877643 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:41.877643 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:41.878852 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:41.878852 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:41.878852 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:41.878852 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:41.878852 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:12:41.878852 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:12:41.878852 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:12:41.878852 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:12:41.878852 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:12:41.878852 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:12:41.878852 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 27 01:12:41.878852 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:41.878852 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 399/815] 10.222.0.42 () {56 vars in 1120 bytes} [Thu Oct 27 01:12:41 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:41.880236 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d83851e-314d-47b3-9557-97fb2bdd3754 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:41.976740 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d83851e-314d-47b3-9557-97fb2bdd3754 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:42.012104 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e2bbd9-ad7d-4230-8b25-22a0d5102a11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:42.012698 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e2bbd9-ad7d-4230-8b25-22a0d5102a11 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:42.012973 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e2bbd9-ad7d-4230-8b25-22a0d5102a11 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:42.013268 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e2bbd9-ad7d-4230-8b25-22a0d5102a11 None admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:42.013918 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e2bbd9-ad7d-4230-8b25-22a0d5102a11 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:42.014213 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e2bbd9-ad7d-4230-8b25-22a0d5102a11 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:42.015568 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e2bbd9-ad7d-4230-8b25-22a0d5102a11 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:42.023553 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 417/816] 10.222.0.42 () {56 vars in 1115 bytes} [Thu Oct 27 01:12:41 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:42.029287 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a16646a-c4fe-47d9-b54e-0040e2040c76 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:42.122649 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e2bbd9-ad7d-4230-8b25-22a0d5102a11 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:42.123394 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:42.126972 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a16646a-c4fe-47d9-b54e-0040e2040c76 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:42.162499 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51dda957-d02e-492d-a74a-508fe5aa415b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:42.163137 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51dda957-d02e-492d-a74a-508fe5aa415b None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:42.163410 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51dda957-d02e-492d-a74a-508fe5aa415b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:42.163675 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51dda957-d02e-492d-a74a-508fe5aa415b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:42.176201 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51dda957-d02e-492d-a74a-508fe5aa415b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:42.177710 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51dda957-d02e-492d-a74a-508fe5aa415b None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:42.193477 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-51dda957-d02e-492d-a74a-508fe5aa415b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:12:42.193477 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:42.193477 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:42.193477 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:42.193477 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:42.193477 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:42.193477 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:42.193477 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:42.193477 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:42.193477 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:42.193477 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:42.193477 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:42.193477 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:12:42.193477 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:12:42.193477 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:12:42.193477 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:12:42.194718 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:42.194718 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:42.194718 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:42.194718 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:42.194718 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:42.194718 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:42.194718 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:42.194718 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:42.194718 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:42.194718 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:42.194718 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:42.194718 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:42.194718 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:42.194718 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:42.194718 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:42.194718 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:42.194718 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:12:42.195991 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:12:42.195991 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:12:42.195991 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:42.195991 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 400/817] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:12:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:42.200431 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51dda957-d02e-492d-a74a-508fe5aa415b None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:42.266684 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:42.302247 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51dda957-d02e-492d-a74a-508fe5aa415b None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:42.337668 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638a9f58-8761-4bbe-b4a7-b162664f71d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:42.338268 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638a9f58-8761-4bbe-b4a7-b162664f71d3 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:42.338544 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638a9f58-8761-4bbe-b4a7-b162664f71d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:42.338821 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638a9f58-8761-4bbe-b4a7-b162664f71d3 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:42.339460 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638a9f58-8761-4bbe-b4a7-b162664f71d3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:42.339758 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638a9f58-8761-4bbe-b4a7-b162664f71d3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:42.341141 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638a9f58-8761-4bbe-b4a7-b162664f71d3 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:42.344238 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e2bbd9-ad7d-4230-8b25-22a0d5102a11 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:42.352407 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 401/818] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:12:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:12:42.358329 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638a9f58-8761-4bbe-b4a7-b162664f71d3 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:42.372527 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:42.383167 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d096946-fc3a-4b11-8292-64f2f03da236 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:42.383795 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d096946-fc3a-4b11-8292-64f2f03da236 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:42.384072 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d096946-fc3a-4b11-8292-64f2f03da236 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:42.384337 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d096946-fc3a-4b11-8292-64f2f03da236 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:42.394434 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d096946-fc3a-4b11-8292-64f2f03da236 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:12:42.399122 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-2d096946-fc3a-4b11-8292-64f2f03da236 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:12:42.399122 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:42.399122 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:42.399122 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:42.399122 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:42.399122 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:42.399122 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:42.399122 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:42.399122 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:42.399122 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:42.399122 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:42.399122 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:42.399122 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:12:42.399122 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:12:42.399122 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:12:42.399122 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:12:42.401161 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:12:42.401161 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:12:42.401161 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:12:42.401161 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:12:42.401161 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:42.401161 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:42.401161 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:42.401161 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:42.401161 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:42.401161 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:42.401161 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:42.401161 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:42.401161 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:42.401161 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:42.401161 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:42.401161 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:42.402142 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:42.402142 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:42.402142 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:42.402142 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:42.402142 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:12:42.402142 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:12:42.402142 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:12:42.402142 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:12:42.402142 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:12:42.402142 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:12:42.402142 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:12:42.402142 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:42.402142 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 418/819] 10.222.0.42 () {56 vars in 1096 bytes} [Thu Oct 27 01:12:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 278 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 27 01:12:42.406238 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d096946-fc3a-4b11-8292-64f2f03da236 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:42.460254 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638a9f58-8761-4bbe-b4a7-b162664f71d3 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:42.485368 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:42.494226 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0913f066-df93-4dd4-9c37-c660640dacf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:42.494851 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0913f066-df93-4dd4-9c37-c660640dacf2 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:42.495129 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0913f066-df93-4dd4-9c37-c660640dacf2 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:42.495408 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0913f066-df93-4dd4-9c37-c660640dacf2 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:42.503079 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0913f066-df93-4dd4-9c37-c660640dacf2 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:42.503706 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d096946-fc3a-4b11-8292-64f2f03da236 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:42.504501 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0913f066-df93-4dd4-9c37-c660640dacf2 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:42.515747 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-0913f066-df93-4dd4-9c37-c660640dacf2 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 27 01:12:42.515747 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:42.515747 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:42.515747 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:42.515747 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:42.515747 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:42.515747 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:42.515747 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:42.515747 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:42.515747 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:42.515747 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:42.515747 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:42.515747 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 27 01:12:42.515747 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 27 01:12:42.515747 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 27 01:12:42.515747 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 27 01:12:42.516841 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:42.516841 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:42.516841 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 27 01:12:42.516841 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:12:42.516841 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 27 01:12:42.516841 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:12:42.516841 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:42.516841 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:42.516841 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:42.516841 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:42.516841 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:42.516841 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:42.516841 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:42.516841 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:42.516841 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:42.516841 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:42.516841 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:42.516841 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:42.518405 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:42.518405 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:42.518405 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 27 01:12:42.518405 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 27 01:12:42.518405 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 27 01:12:42.518405 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 27 01:12:42.518405 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 27 01:12:42.518405 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 27 01:12:42.518405 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 27 01:12:42.518405 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:42.518405 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 402/820] 10.222.0.42 () {56 vars in 1093 bytes} [Thu Oct 27 01:12:42 2022] GET /identity/v3/users/nova => generated 82 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:42.523674 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0913f066-df93-4dd4-9c37-c660640dacf2 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:42.539502 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80699dfc-0689-4ad0-aac7-8c01f2134bce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:42.540097 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80699dfc-0689-4ad0-aac7-8c01f2134bce None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:42.540372 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80699dfc-0689-4ad0-aac7-8c01f2134bce None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:42.540687 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80699dfc-0689-4ad0-aac7-8c01f2134bce None admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:42.541394 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80699dfc-0689-4ad0-aac7-8c01f2134bce None admin] RBAC: Adding query filter params (name=admin) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:42.541677 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80699dfc-0689-4ad0-aac7-8c01f2134bce None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:42.543010 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80699dfc-0689-4ad0-aac7-8c01f2134bce None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:42.551256 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 419/821] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:12:42 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:12:42.557120 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80699dfc-0689-4ad0-aac7-8c01f2134bce None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:42.591899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:42.623229 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0913f066-df93-4dd4-9c37-c660640dacf2 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:42.653373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80699dfc-0689-4ad0-aac7-8c01f2134bce None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:42.656692 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6cb3eba-bd03-4890-a262-8c0dd8586306 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:42.657326 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cb3eba-bd03-4890-a262-8c0dd8586306 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:42.657607 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cb3eba-bd03-4890-a262-8c0dd8586306 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:42.657879 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cb3eba-bd03-4890-a262-8c0dd8586306 None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:42.659190 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6cb3eba-bd03-4890-a262-8c0dd8586306 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:42.659492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6cb3eba-bd03-4890-a262-8c0dd8586306 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:42.660925 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6cb3eba-bd03-4890-a262-8c0dd8586306 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:42.678504 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 403/822] 10.222.0.42 () {56 vars in 1133 bytes} [Thu Oct 27 01:12:42 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:12:42.684438 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6cb3eba-bd03-4890-a262-8c0dd8586306 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:42.687651 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d0248e-11fe-4b1d-9693-0f1d6608b9cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:42.688276 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d0248e-11fe-4b1d-9693-0f1d6608b9cf None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:42.688554 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d0248e-11fe-4b1d-9693-0f1d6608b9cf None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:42.688830 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d0248e-11fe-4b1d-9693-0f1d6608b9cf None admin] PATH_INFO: `/v3/users/b04c4b80d6df43a9bd761838dcea3992` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:42.702564 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d0248e-11fe-4b1d-9693-0f1d6608b9cf None admin] RBAC: Authorizing `identity:get_user(user_id=b04c4b80d6df43a9bd761838dcea3992)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:42.703978 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d0248e-11fe-4b1d-9693-0f1d6608b9cf None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:42.706097 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 420/823] 10.222.0.42 () {56 vars in 1177 bytes} [Thu Oct 27 01:12:42 2022] GET /identity/v3/users/b04c4b80d6df43a9bd761838dcea3992 => generated 285 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:12:42.711739 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d0248e-11fe-4b1d-9693-0f1d6608b9cf None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:42.714212 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:42.781220 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6cb3eba-bd03-4890-a262-8c0dd8586306 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:42.808882 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:42.811279 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d0248e-11fe-4b1d-9693-0f1d6608b9cf None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:42.818727 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f63a48a-421a-417e-9307-389d9259581d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:42.819337 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f63a48a-421a-417e-9307-389d9259581d None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:42.819612 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f63a48a-421a-417e-9307-389d9259581d None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:42.819882 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f63a48a-421a-417e-9307-389d9259581d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:42.844069 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a0cbdd-8012-47be-8e7c-1c1130bb24ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:42.844723 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a0cbdd-8012-47be-8e7c-1c1130bb24ac None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:42.844998 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a0cbdd-8012-47be-8e7c-1c1130bb24ac None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:42.845298 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a0cbdd-8012-47be-8e7c-1c1130bb24ac None admin] PATH_INFO: `/v3/projects/441c0bff70a0478ea24739ab47979b0e` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:42.847002 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a0cbdd-8012-47be-8e7c-1c1130bb24ac None admin] RBAC: Authorizing `identity:get_project(project_id=441c0bff70a0478ea24739ab47979b0e)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:42.848426 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a0cbdd-8012-47be-8e7c-1c1130bb24ac None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:42.850068 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 421/824] 10.222.0.42 () {56 vars in 1186 bytes} [Thu Oct 27 01:12:42 2022] GET /identity/v3/projects/441c0bff70a0478ea24739ab47979b0e => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:12:42.855955 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a0cbdd-8012-47be-8e7c-1c1130bb24ac None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:42.887697 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f63a48a-421a-417e-9307-389d9259581d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:42.889185 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f63a48a-421a-417e-9307-389d9259581d None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:42.924495 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:42.956103 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a0cbdd-8012-47be-8e7c-1c1130bb24ac None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:42.985789 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 404/825] 10.222.0.42 () {58 vars in 1282 bytes} [Thu Oct 27 01:12:42 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 27 01:12:42.991515 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f63a48a-421a-417e-9307-389d9259581d None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:42.993147 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5344e6-89a2-4e5b-877d-d8937e6e7b58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:42.993804 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5344e6-89a2-4e5b-877d-d8937e6e7b58 None admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:42.994085 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5344e6-89a2-4e5b-877d-d8937e6e7b58 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:42.994358 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5344e6-89a2-4e5b-877d-d8937e6e7b58 None admin] PATH_INFO: `/v3/projects/441c0bff70a0478ea24739ab47979b0e/users/b04c4b80d6df43a9bd761838dcea3992/roles/414286927fdd44688107ff5e10235acf` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:42.996243 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5344e6-89a2-4e5b-877d-d8937e6e7b58 None admin] RBAC: Authorizing `identity:create_grant(project_id=441c0bff70a0478ea24739ab47979b0e, user_id=b04c4b80d6df43a9bd761838dcea3992, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:42.997692 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5344e6-89a2-4e5b-877d-d8937e6e7b58 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:43.011370 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 422/826] 10.222.0.42 () {58 vars in 1439 bytes} [Thu Oct 27 01:12:42 2022] PUT /identity/v3/projects/441c0bff70a0478ea24739ab47979b0e/users/b04c4b80d6df43a9bd761838dcea3992/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:12:43.019801 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:43.099166 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f63a48a-421a-417e-9307-389d9259581d None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:43.135958 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2b2ddf-8476-4d32-b7ed-3902a9e51bf6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:43.136594 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2b2ddf-8476-4d32-b7ed-3902a9e51bf6 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:43.136870 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2b2ddf-8476-4d32-b7ed-3902a9e51bf6 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:43.137165 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2b2ddf-8476-4d32-b7ed-3902a9e51bf6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:43.148907 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2b2ddf-8476-4d32-b7ed-3902a9e51bf6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:43.150386 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2b2ddf-8476-4d32-b7ed-3902a9e51bf6 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:43.166433 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-df2b2ddf-8476-4d32-b7ed-3902a9e51bf6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:12:43.166433 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:43.166433 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:43.166433 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:43.166433 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:43.166433 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:43.166433 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:43.166433 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:43.166433 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:43.166433 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:43.166433 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:43.166433 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:43.166433 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:12:43.166433 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:12:43.166433 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:12:43.166433 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:12:43.167442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:43.167442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:43.167442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:43.167442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:43.167442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:43.167442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:43.167442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:43.167442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:43.167442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:43.167442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:43.167442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:43.167442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:43.167442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:43.167442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:43.167442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:43.167442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:43.167442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:12:43.168953 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:12:43.168953 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:12:43.168953 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:43.168953 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 405/827] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:12:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 27 01:12:43.173351 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5344e6-89a2-4e5b-877d-d8937e6e7b58 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:43.274038 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5344e6-89a2-4e5b-877d-d8937e6e7b58 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:43.302233 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:43.309196 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d12721-413e-4beb-bf2b-ecd8f32ec1c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:43.309983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d12721-413e-4beb-bf2b-ecd8f32ec1c7 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:43.310409 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d12721-413e-4beb-bf2b-ecd8f32ec1c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:43.310706 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d12721-413e-4beb-bf2b-ecd8f32ec1c7 None admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:43.311343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d12721-413e-4beb-bf2b-ecd8f32ec1c7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:43.311644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d12721-413e-4beb-bf2b-ecd8f32ec1c7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:43.313008 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d12721-413e-4beb-bf2b-ecd8f32ec1c7 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:43.324804 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 423/828] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:12:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:43.330386 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2b2ddf-8476-4d32-b7ed-3902a9e51bf6 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:43.408139 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:43.427213 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2b2ddf-8476-4d32-b7ed-3902a9e51bf6 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:43.459875 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd43429-e056-4684-9cb1-cafe9c7792b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:43.460488 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd43429-e056-4684-9cb1-cafe9c7792b3 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:43.460758 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd43429-e056-4684-9cb1-cafe9c7792b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:43.461020 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd43429-e056-4684-9cb1-cafe9c7792b3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:43.467655 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd43429-e056-4684-9cb1-cafe9c7792b3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:43.469104 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd43429-e056-4684-9cb1-cafe9c7792b3 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:43.480610 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-7cd43429-e056-4684-9cb1-cafe9c7792b3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:12:43.480610 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:43.480610 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:43.480610 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:43.480610 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:43.480610 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:43.480610 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:43.480610 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:43.480610 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:43.480610 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:43.480610 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:43.480610 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:43.480610 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 27 01:12:43.480610 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 27 01:12:43.480610 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 27 01:12:43.481597 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 27 01:12:43.481597 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:43.481597 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:43.481597 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:43.481597 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:43.481597 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:43.481597 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:43.481597 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:43.481597 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:43.481597 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:43.481597 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:43.481597 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:43.481597 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:43.481597 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:43.481597 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:43.481597 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:43.481597 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:43.482509 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 27 01:12:43.482509 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 27 01:12:43.482509 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 27 01:12:43.482509 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 27 01:12:43.482509 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 27 01:12:43.482509 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 27 01:12:43.482509 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:12:43.482509 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:12:43.482509 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 406/829] 10.222.0.42 () {56 vars in 1111 bytes} [Thu Oct 27 01:12:43 2022] GET /identity/v3/projects/service => generated 88 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:43.487747 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d12721-413e-4beb-bf2b-ecd8f32ec1c7 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:43.584597 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d12721-413e-4beb-bf2b-ecd8f32ec1c7 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:43.616275 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bae5069-ef4f-49ce-8776-413165bae88f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:43.616862 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bae5069-ef4f-49ce-8776-413165bae88f None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:43.617164 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bae5069-ef4f-49ce-8776-413165bae88f None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:43.617432 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bae5069-ef4f-49ce-8776-413165bae88f None admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:43.618106 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bae5069-ef4f-49ce-8776-413165bae88f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:43.618400 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bae5069-ef4f-49ce-8776-413165bae88f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:43.619802 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bae5069-ef4f-49ce-8776-413165bae88f None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:43.632303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 424/830] 10.222.0.42 () {56 vars in 1148 bytes} [Thu Oct 27 01:12:43 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:43.638211 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd43429-e056-4684-9cb1-cafe9c7792b3 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:43.739305 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd43429-e056-4684-9cb1-cafe9c7792b3 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:43.772693 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f0c8e0-944e-435f-b9a7-c87f82370033 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:43.773324 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f0c8e0-944e-435f-b9a7-c87f82370033 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:43.773601 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f0c8e0-944e-435f-b9a7-c87f82370033 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:43.773871 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f0c8e0-944e-435f-b9a7-c87f82370033 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:43.774586 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f0c8e0-944e-435f-b9a7-c87f82370033 None admin] RBAC: Adding query filter params (role.id=04d145c26220455b969c13d2d24e84e8, scope.project.id=23aad848d47c4d27ad4c138ff869e40f, user.id=51033d56f85f4282a49ded6338440677) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:43.774889 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f0c8e0-944e-435f-b9a7-c87f82370033 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:43.776299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f0c8e0-944e-435f-b9a7-c87f82370033 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:43.781859 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 407/831] 10.222.0.42 () {56 vars in 1374 bytes} [Thu Oct 27 01:12:43 2022] GET /identity/v3/role_assignments?user.id=51033d56f85f4282a49ded6338440677&scope.project.id=23aad848d47c4d27ad4c138ff869e40f&role.id=04d145c26220455b969c13d2d24e84e8 => generated 608 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:45.143715 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba422121-2f16-4574-afa6-53cd5d38a878 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:45.144231 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba422121-2f16-4574-afa6-53cd5d38a878 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:45.144231 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba422121-2f16-4574-afa6-53cd5d38a878 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:45.145551 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 425/832] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:45.151374 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23633fa-b14c-4c37-83f7-138390d67b60 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:45.151811 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23633fa-b14c-4c37-83f7-138390d67b60 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:45.151907 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23633fa-b14c-4c37-83f7-138390d67b60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:45.159162 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:45.170818 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-a23633fa-b14c-4c37-83f7-138390d67b60 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:45.172337 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-a23633fa-b14c-4c37-83f7-138390d67b60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:45.236704 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 408/833] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:45 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:12:45.506544 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba422121-2f16-4574-afa6-53cd5d38a878 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:45.555878 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:45.604970 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba422121-2f16-4574-afa6-53cd5d38a878 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:45.638477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cdbdc3a-6632-477f-a213-4c6e9f3a9a10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:45.639088 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cdbdc3a-6632-477f-a213-4c6e9f3a9a10 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:45.639365 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cdbdc3a-6632-477f-a213-4c6e9f3a9a10 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:45.639636 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cdbdc3a-6632-477f-a213-4c6e9f3a9a10 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:45.649447 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cdbdc3a-6632-477f-a213-4c6e9f3a9a10 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:12:45.654212 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-7cdbdc3a-6632-477f-a213-4c6e9f3a9a10 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:12:45.654212 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:12:45.654212 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:12:45.654212 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:12:45.654212 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:12:45.654212 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:12:45.654212 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:12:45.654212 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:12:45.654212 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:12:45.654212 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:12:45.654212 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:12:45.654212 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:12:45.654212 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:12:45.654212 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:12:45.654212 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:12:45.654212 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:12:45.655274 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:12:45.655274 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:12:45.655274 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:12:45.655274 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:12:45.655274 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:12:45.655274 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:12:45.655274 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:12:45.655274 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:12:45.655274 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:12:45.655274 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:12:45.655274 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:12:45.655274 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:12:45.655274 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:12:45.655274 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:12:45.655274 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:12:45.655274 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:12:45.656426 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:12:45.656426 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:12:45.656426 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:12:45.656426 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:12:45.656426 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:12:45.656426 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:12:45.656426 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:12:45.656426 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:12:45.656426 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:12:45.656426 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:12:45.656426 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:12:45.656426 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:12:45.656426 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 426/834] 10.222.0.42 () {56 vars in 1096 bytes} [Thu Oct 27 01:12:45 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:12:45.661396 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23633fa-b14c-4c37-83f7-138390d67b60 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:45.754983 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23633fa-b14c-4c37-83f7-138390d67b60 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:45.787267 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3270253-7ed8-441d-8822-3b8c75aba773 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:45.787792 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3270253-7ed8-441d-8822-3b8c75aba773 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:45.788063 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3270253-7ed8-441d-8822-3b8c75aba773 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:45.788329 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3270253-7ed8-441d-8822-3b8c75aba773 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:45.788976 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3270253-7ed8-441d-8822-3b8c75aba773 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:45.789299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3270253-7ed8-441d-8822-3b8c75aba773 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:45.790639 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3270253-7ed8-441d-8822-3b8c75aba773 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:45.791993 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:45.799933 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 409/835] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:12:45 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:45.805795 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cdbdc3a-6632-477f-a213-4c6e9f3a9a10 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:45.901214 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cdbdc3a-6632-477f-a213-4c6e9f3a9a10 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:45.911965 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:45.935335 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9215c3cd-f8a1-4c6a-b8de-058bc817f348 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:45.935952 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9215c3cd-f8a1-4c6a-b8de-058bc817f348 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:45.936227 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9215c3cd-f8a1-4c6a-b8de-058bc817f348 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:45.936495 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9215c3cd-f8a1-4c6a-b8de-058bc817f348 None admin] PATH_INFO: `/v3/users/b04c4b80d6df43a9bd761838dcea3992` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:45.937621 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9215c3cd-f8a1-4c6a-b8de-058bc817f348 None admin] RBAC: Authorizing `identity:get_user(user_id=b04c4b80d6df43a9bd761838dcea3992)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:45.939038 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9215c3cd-f8a1-4c6a-b8de-058bc817f348 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:45.940775 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 427/836] 10.222.0.42 () {56 vars in 1177 bytes} [Thu Oct 27 01:12:45 2022] GET /identity/v3/users/b04c4b80d6df43a9bd761838dcea3992 => generated 285 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:45.946485 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3270253-7ed8-441d-8822-3b8c75aba773 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:46.043124 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3270253-7ed8-441d-8822-3b8c75aba773 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:46.075692 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ecc17e8-de8d-4ff7-91b4-e8c6151b9e43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:46.076285 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ecc17e8-de8d-4ff7-91b4-e8c6151b9e43 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:46.076562 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ecc17e8-de8d-4ff7-91b4-e8c6151b9e43 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:46.076829 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ecc17e8-de8d-4ff7-91b4-e8c6151b9e43 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:46.133887 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:46.140468 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ecc17e8-de8d-4ff7-91b4-e8c6151b9e43 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:46.141920 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ecc17e8-de8d-4ff7-91b4-e8c6151b9e43 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:46.235650 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 410/837] 10.222.0.42 () {58 vars in 1282 bytes} [Thu Oct 27 01:12:45 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 27 01:12:46.241705 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9215c3cd-f8a1-4c6a-b8de-058bc817f348 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:46.338857 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9215c3cd-f8a1-4c6a-b8de-058bc817f348 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:46.371836 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-839b4a80-832a-457f-b1f0-e6fd5cdebcca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:46.372451 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839b4a80-832a-457f-b1f0-e6fd5cdebcca None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:46.372726 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839b4a80-832a-457f-b1f0-e6fd5cdebcca None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:46.373016 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839b4a80-832a-457f-b1f0-e6fd5cdebcca None admin] PATH_INFO: `/v3/projects/441c0bff70a0478ea24739ab47979b0e` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:46.374144 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-839b4a80-832a-457f-b1f0-e6fd5cdebcca None admin] RBAC: Authorizing `identity:get_project(project_id=441c0bff70a0478ea24739ab47979b0e)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:46.375582 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-839b4a80-832a-457f-b1f0-e6fd5cdebcca None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:46.377298 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 428/838] 10.222.0.42 () {56 vars in 1186 bytes} [Thu Oct 27 01:12:46 2022] GET /identity/v3/projects/441c0bff70a0478ea24739ab47979b0e => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:46.383382 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ecc17e8-de8d-4ff7-91b4-e8c6151b9e43 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:46.473598 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:46.479077 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ecc17e8-de8d-4ff7-91b4-e8c6151b9e43 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:46.512403 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edece51-8b21-4c25-baca-da600dddb7df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:46.512986 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edece51-8b21-4c25-baca-da600dddb7df None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:46.513285 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edece51-8b21-4c25-baca-da600dddb7df None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:46.513551 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edece51-8b21-4c25-baca-da600dddb7df None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:46.514252 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1edece51-8b21-4c25-baca-da600dddb7df None admin] RBAC: Adding query filter params (role.id=414286927fdd44688107ff5e10235acf, scope.project.id=441c0bff70a0478ea24739ab47979b0e, user.id=b04c4b80d6df43a9bd761838dcea3992) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:46.514552 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1edece51-8b21-4c25-baca-da600dddb7df None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:46.515958 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1edece51-8b21-4c25-baca-da600dddb7df None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:46.522456 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 411/839] 10.222.0.42 () {56 vars in 1374 bytes} [Thu Oct 27 01:12:46 2022] GET /identity/v3/role_assignments?user.id=b04c4b80d6df43a9bd761838dcea3992&scope.project.id=441c0bff70a0478ea24739ab47979b0e&role.id=414286927fdd44688107ff5e10235acf => generated 608 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:48.665847 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea64f192-dfe9-470d-a3bd-a2912c70d186 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:48.666377 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea64f192-dfe9-470d-a3bd-a2912c70d186 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:48.666377 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea64f192-dfe9-470d-a3bd-a2912c70d186 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:48.667677 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 429/840] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:48.672778 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d577bf-14c6-4ded-b516-e93068cd3512 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:48.673220 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d577bf-14c6-4ded-b516-e93068cd3512 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:48.673346 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d577bf-14c6-4ded-b516-e93068cd3512 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:48.692424 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-88d577bf-14c6-4ded-b516-e93068cd3512 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:48.693637 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-88d577bf-14c6-4ded-b516-e93068cd3512 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:48.757368 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 412/841] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:48 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:12:49.023262 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea64f192-dfe9-470d-a3bd-a2912c70d186 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:49.069968 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:49.121419 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea64f192-dfe9-470d-a3bd-a2912c70d186 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:49.153149 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676b1e04-d482-4f80-9f2b-9577a463d131 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:49.153757 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676b1e04-d482-4f80-9f2b-9577a463d131 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:49.154024 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676b1e04-d482-4f80-9f2b-9577a463d131 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:49.154286 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676b1e04-d482-4f80-9f2b-9577a463d131 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:49.155364 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-676b1e04-d482-4f80-9f2b-9577a463d131 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:49.156761 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-676b1e04-d482-4f80-9f2b-9577a463d131 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:49.158358 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 430/842] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:12:49 2022] GET /identity/v3/domains/default => generated 199 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:49.164150 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d577bf-14c6-4ded-b516-e93068cd3512 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:49.250048 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:49.259888 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d577bf-14c6-4ded-b516-e93068cd3512 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:49.292719 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9c426e-98eb-4a74-9dda-7b58152a8dc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:49.293279 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9c426e-98eb-4a74-9dda-7b58152a8dc4 None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:49.293550 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9c426e-98eb-4a74-9dda-7b58152a8dc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:49.293814 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9c426e-98eb-4a74-9dda-7b58152a8dc4 None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:49.294444 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e9c426e-98eb-4a74-9dda-7b58152a8dc4 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:49.295848 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e9c426e-98eb-4a74-9dda-7b58152a8dc4 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:49.317914 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 413/843] 10.222.0.42 () {60 vars in 1132 bytes} [Thu Oct 27 01:12:49 2022] POST /identity/v3/users => generated 286 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:12:51.495481 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c48341-f783-4a6c-892e-d7bd5b2f72c1 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:51.496017 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c48341-f783-4a6c-892e-d7bd5b2f72c1 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:51.496017 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c48341-f783-4a6c-892e-d7bd5b2f72c1 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:51.497360 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 431/844] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:51.503349 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3051a48-cad2-404f-af65-5719ca6008f0 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:51.503753 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3051a48-cad2-404f-af65-5719ca6008f0 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:51.503888 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3051a48-cad2-404f-af65-5719ca6008f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:51.521796 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-e3051a48-cad2-404f-af65-5719ca6008f0 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:51.522972 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e3051a48-cad2-404f-af65-5719ca6008f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:51.586858 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 414/845] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:51 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:12:51.856897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c48341-f783-4a6c-892e-d7bd5b2f72c1 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:51.954272 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c48341-f783-4a6c-892e-d7bd5b2f72c1 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:51.967244 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:51.986952 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a93f63-0c6d-4e09-84ca-9bc3c6c889c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:51.987556 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a93f63-0c6d-4e09-84ca-9bc3c6c889c6 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:51.987827 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a93f63-0c6d-4e09-84ca-9bc3c6c889c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:51.988088 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a93f63-0c6d-4e09-84ca-9bc3c6c889c6 None admin] PATH_INFO: `/v3/roles/bb9bd8ef24ff4aa5b639a2bd8567345b` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:51.989482 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a93f63-0c6d-4e09-84ca-9bc3c6c889c6 None admin] RBAC: Authorizing `identity:get_role(role_id=bb9bd8ef24ff4aa5b639a2bd8567345b)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:51.990819 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a93f63-0c6d-4e09-84ca-9bc3c6c889c6 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:51.992092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 432/846] 10.222.0.42 () {56 vars in 1177 bytes} [Thu Oct 27 01:12:51 2022] GET /identity/v3/roles/bb9bd8ef24ff4aa5b639a2bd8567345b => generated 223 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:51.997815 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3051a48-cad2-404f-af65-5719ca6008f0 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:52.073011 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:52.095773 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3051a48-cad2-404f-af65-5719ca6008f0 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:52.130050 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d1ce31-0795-4e2a-858f-38451fa21189 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:52.130668 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d1ce31-0795-4e2a-858f-38451fa21189 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:52.130945 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d1ce31-0795-4e2a-858f-38451fa21189 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:52.131215 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d1ce31-0795-4e2a-858f-38451fa21189 None admin] PATH_INFO: `/v3/users/eb2423169d4b49c38ee8e6080250cd37` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:52.143917 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d1ce31-0795-4e2a-858f-38451fa21189 None admin] RBAC: Authorizing `identity:get_user(user_id=eb2423169d4b49c38ee8e6080250cd37)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:52.145362 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d1ce31-0795-4e2a-858f-38451fa21189 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:52.146941 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 415/847] 10.222.0.42 () {56 vars in 1177 bytes} [Thu Oct 27 01:12:51 2022] GET /identity/v3/users/eb2423169d4b49c38ee8e6080250cd37 => generated 286 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:52.152702 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a93f63-0c6d-4e09-84ca-9bc3c6c889c6 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:52.248898 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a93f63-0c6d-4e09-84ca-9bc3c6c889c6 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:52.281902 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f93773b-0049-49c3-a6e5-3a912f83179f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:52.282514 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f93773b-0049-49c3-a6e5-3a912f83179f None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:52.282814 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f93773b-0049-49c3-a6e5-3a912f83179f None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:52.283167 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f93773b-0049-49c3-a6e5-3a912f83179f None admin] PATH_INFO: `/v3/projects/441c0bff70a0478ea24739ab47979b0e` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:52.284277 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f93773b-0049-49c3-a6e5-3a912f83179f None admin] RBAC: Authorizing `identity:get_project(project_id=441c0bff70a0478ea24739ab47979b0e)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:52.285722 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f93773b-0049-49c3-a6e5-3a912f83179f None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:52.287310 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 433/848] 10.222.0.42 () {56 vars in 1186 bytes} [Thu Oct 27 01:12:52 2022] GET /identity/v3/projects/441c0bff70a0478ea24739ab47979b0e => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:52.293215 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d1ce31-0795-4e2a-858f-38451fa21189 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:52.388342 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d1ce31-0795-4e2a-858f-38451fa21189 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:52.420218 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d497627e-4078-4d60-97fd-83323beea09e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:52.420860 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d497627e-4078-4d60-97fd-83323beea09e None admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:52.421169 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d497627e-4078-4d60-97fd-83323beea09e None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:52.421441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d497627e-4078-4d60-97fd-83323beea09e None admin] PATH_INFO: `/v3/projects/441c0bff70a0478ea24739ab47979b0e/users/eb2423169d4b49c38ee8e6080250cd37/roles/bb9bd8ef24ff4aa5b639a2bd8567345b` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:52.424085 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d497627e-4078-4d60-97fd-83323beea09e None admin] RBAC: Authorizing `identity:create_grant(project_id=441c0bff70a0478ea24739ab47979b0e, user_id=eb2423169d4b49c38ee8e6080250cd37, role_id=bb9bd8ef24ff4aa5b639a2bd8567345b)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:52.425724 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d497627e-4078-4d60-97fd-83323beea09e None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:52.427833 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:52.439743 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 416/849] 10.222.0.42 () {58 vars in 1439 bytes} [Thu Oct 27 01:12:52 2022] PUT /identity/v3/projects/441c0bff70a0478ea24739ab47979b0e/users/eb2423169d4b49c38ee8e6080250cd37/roles/bb9bd8ef24ff4aa5b639a2bd8567345b => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:12:54.560610 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c75c9f-d114-4e9f-84c1-96c0419982ba None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:54.561174 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c75c9f-d114-4e9f-84c1-96c0419982ba None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:54.561174 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c75c9f-d114-4e9f-84c1-96c0419982ba None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:54.562496 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 434/850] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:54.567787 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db0e591-ebca-48e5-9e50-dc2b4e16f93e None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:54.568209 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db0e591-ebca-48e5-9e50-dc2b4e16f93e None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:54.568317 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db0e591-ebca-48e5-9e50-dc2b4e16f93e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:54.587462 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-4db0e591-ebca-48e5-9e50-dc2b4e16f93e None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:54.588627 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-4db0e591-ebca-48e5-9e50-dc2b4e16f93e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:54.771286 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 417/851] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:54 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:12:55.040386 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c75c9f-d114-4e9f-84c1-96c0419982ba None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:55.090007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:55.140123 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c75c9f-d114-4e9f-84c1-96c0419982ba None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:55.172247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2cf376d-5fcb-4cd4-9698-5ecf59de2e91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:55.172851 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2cf376d-5fcb-4cd4-9698-5ecf59de2e91 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:55.173647 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2cf376d-5fcb-4cd4-9698-5ecf59de2e91 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:55.174461 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2cf376d-5fcb-4cd4-9698-5ecf59de2e91 None admin] PATH_INFO: `/v3/roles/bb9bd8ef24ff4aa5b639a2bd8567345b` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:55.175835 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2cf376d-5fcb-4cd4-9698-5ecf59de2e91 None admin] RBAC: Authorizing `identity:get_role(role_id=bb9bd8ef24ff4aa5b639a2bd8567345b)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:55.177207 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2cf376d-5fcb-4cd4-9698-5ecf59de2e91 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:55.178496 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 435/852] 10.222.0.42 () {56 vars in 1177 bytes} [Thu Oct 27 01:12:55 2022] GET /identity/v3/roles/bb9bd8ef24ff4aa5b639a2bd8567345b => generated 223 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:55.184596 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db0e591-ebca-48e5-9e50-dc2b4e16f93e None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:55.279311 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db0e591-ebca-48e5-9e50-dc2b4e16f93e None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:55.302167 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:55.313081 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-603daab6-ebae-4ece-bfbf-5d3eae9653cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:55.313690 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603daab6-ebae-4ece-bfbf-5d3eae9653cd None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:55.314914 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603daab6-ebae-4ece-bfbf-5d3eae9653cd None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:55.315187 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603daab6-ebae-4ece-bfbf-5d3eae9653cd None admin] PATH_INFO: `/v3/users/eb2423169d4b49c38ee8e6080250cd37` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:55.316289 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-603daab6-ebae-4ece-bfbf-5d3eae9653cd None admin] RBAC: Authorizing `identity:get_user(user_id=eb2423169d4b49c38ee8e6080250cd37)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:55.317750 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-603daab6-ebae-4ece-bfbf-5d3eae9653cd None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:55.319272 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 418/853] 10.222.0.42 () {56 vars in 1177 bytes} [Thu Oct 27 01:12:55 2022] GET /identity/v3/users/eb2423169d4b49c38ee8e6080250cd37 => generated 286 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:55.324973 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2cf376d-5fcb-4cd4-9698-5ecf59de2e91 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:55.422762 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2cf376d-5fcb-4cd4-9698-5ecf59de2e91 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:55.440343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:55.456736 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c29382-9976-4c87-ba02-f1faed59f06d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:55.457363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c29382-9976-4c87-ba02-f1faed59f06d None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:55.457637 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c29382-9976-4c87-ba02-f1faed59f06d None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:55.457904 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c29382-9976-4c87-ba02-f1faed59f06d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:55.522383 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c29382-9976-4c87-ba02-f1faed59f06d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:55.523807 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c29382-9976-4c87-ba02-f1faed59f06d None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:55.617139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 436/854] 10.222.0.42 () {58 vars in 1282 bytes} [Thu Oct 27 01:12:55 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 27 01:12:55.623060 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-603daab6-ebae-4ece-bfbf-5d3eae9653cd None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:55.717971 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-603daab6-ebae-4ece-bfbf-5d3eae9653cd None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:55.750056 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ae8be4-fb5f-4b63-a5d9-e22cfecf0343 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:55.750660 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ae8be4-fb5f-4b63-a5d9-e22cfecf0343 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:55.750933 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ae8be4-fb5f-4b63-a5d9-e22cfecf0343 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:55.751200 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ae8be4-fb5f-4b63-a5d9-e22cfecf0343 None admin] PATH_INFO: `/v3/projects/441c0bff70a0478ea24739ab47979b0e` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:55.752337 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ae8be4-fb5f-4b63-a5d9-e22cfecf0343 None admin] RBAC: Authorizing `identity:get_project(project_id=441c0bff70a0478ea24739ab47979b0e)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:55.753767 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ae8be4-fb5f-4b63-a5d9-e22cfecf0343 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:55.755339 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 419/855] 10.222.0.42 () {56 vars in 1186 bytes} [Thu Oct 27 01:12:55 2022] GET /identity/v3/projects/441c0bff70a0478ea24739ab47979b0e => generated 311 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:55.761678 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c29382-9976-4c87-ba02-f1faed59f06d None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:55.779213 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:55.857728 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c29382-9976-4c87-ba02-f1faed59f06d None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:55.889318 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd081995-39e5-4612-a419-222cd8d185b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:55.889898 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd081995-39e5-4612-a419-222cd8d185b5 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:55.890168 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd081995-39e5-4612-a419-222cd8d185b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:55.890433 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd081995-39e5-4612-a419-222cd8d185b5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:55.891136 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd081995-39e5-4612-a419-222cd8d185b5 None admin] RBAC: Adding query filter params (role.id=bb9bd8ef24ff4aa5b639a2bd8567345b, scope.project.id=441c0bff70a0478ea24739ab47979b0e, user.id=eb2423169d4b49c38ee8e6080250cd37) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:12:55.891438 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd081995-39e5-4612-a419-222cd8d185b5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:55.893395 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd081995-39e5-4612-a419-222cd8d185b5 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:55.898812 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 437/856] 10.222.0.42 () {56 vars in 1374 bytes} [Thu Oct 27 01:12:55 2022] GET /identity/v3/role_assignments?user.id=eb2423169d4b49c38ee8e6080250cd37&scope.project.id=441c0bff70a0478ea24739ab47979b0e&role.id=bb9bd8ef24ff4aa5b639a2bd8567345b => generated 608 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:58.031027 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5ce024-4a0b-4365-9dc2-b98cc57cbe6f None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:58.031575 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5ce024-4a0b-4365-9dc2-b98cc57cbe6f None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:58.031575 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5ce024-4a0b-4365-9dc2-b98cc57cbe6f None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:58.033651 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 420/857] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:12:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:12:58.038922 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8e5da0-dfc0-4801-afe0-d2a46dceceff None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:58.039356 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8e5da0-dfc0-4801-afe0-d2a46dceceff None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:58.039495 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8e5da0-dfc0-4801-afe0-d2a46dceceff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:58.058176 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-fa8e5da0-dfc0-4801-afe0-d2a46dceceff None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:12:58.059351 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-fa8e5da0-dfc0-4801-afe0-d2a46dceceff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:12:58.123245 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 438/858] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:12:58 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:12:58.397101 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5ce024-4a0b-4365-9dc2-b98cc57cbe6f None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:58.456284 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:58.496329 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5ce024-4a0b-4365-9dc2-b98cc57cbe6f None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:58.528501 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be62f2f-cdd9-4e7b-872a-777ad221a872 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:58.529128 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be62f2f-cdd9-4e7b-872a-777ad221a872 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:58.529535 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be62f2f-cdd9-4e7b-872a-777ad221a872 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:58.529798 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be62f2f-cdd9-4e7b-872a-777ad221a872 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:58.530877 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be62f2f-cdd9-4e7b-872a-777ad221a872 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:58.532277 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be62f2f-cdd9-4e7b-872a-777ad221a872 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:58.534836 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 421/859] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:12:58 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:12:58.540613 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8e5da0-dfc0-4801-afe0-d2a46dceceff None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:12:58.555158 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:12:58.637953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8e5da0-dfc0-4801-afe0-d2a46dceceff None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:12:58.670918 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9f4ad7-086b-4e48-bc1e-2ca54bbfcec0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:12:58.671510 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9f4ad7-086b-4e48-bc1e-2ca54bbfcec0 None admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:12:58.671784 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9f4ad7-086b-4e48-bc1e-2ca54bbfcec0 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:12:58.672051 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9f4ad7-086b-4e48-bc1e-2ca54bbfcec0 None admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:12:58.672689 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa9f4ad7-086b-4e48-bc1e-2ca54bbfcec0 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:12:58.674702 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa9f4ad7-086b-4e48-bc1e-2ca54bbfcec0 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:12:58.699951 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 439/860] 10.222.0.42 () {60 vars in 1141 bytes} [Thu Oct 27 01:12:58 2022] POST /identity/v3/projects => generated 311 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:13:00.853691 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e62615b-64e5-4a37-a1a7-4bc64c5d81a2 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:00.854155 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e62615b-64e5-4a37-a1a7-4bc64c5d81a2 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:00.854245 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e62615b-64e5-4a37-a1a7-4bc64c5d81a2 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:00.855586 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 422/861] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:13:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:13:00.860988 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6981fbfa-1bf9-4e38-9a4f-c37c4e7c321d None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:00.861518 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6981fbfa-1bf9-4e38-9a4f-c37c4e7c321d None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:00.861518 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6981fbfa-1bf9-4e38-9a4f-c37c4e7c321d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:00.879489 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-6981fbfa-1bf9-4e38-9a4f-c37c4e7c321d None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:13:00.880674 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-6981fbfa-1bf9-4e38-9a4f-c37c4e7c321d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:13:00.937912 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:13:00.947422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 440/862] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:13:00 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:13:01.217987 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e62615b-64e5-4a37-a1a7-4bc64c5d81a2 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:13:01.316416 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e62615b-64e5-4a37-a1a7-4bc64c5d81a2 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:13:01.341250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:13:01.349453 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480c2200-fc7a-446e-bb11-b5371b142eaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:13:01.350068 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480c2200-fc7a-446e-bb11-b5371b142eaf None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:01.350346 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480c2200-fc7a-446e-bb11-b5371b142eaf None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:01.350618 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480c2200-fc7a-446e-bb11-b5371b142eaf None admin] PATH_INFO: `/v3/domains/default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:01.351716 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480c2200-fc7a-446e-bb11-b5371b142eaf None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:13:01.353170 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480c2200-fc7a-446e-bb11-b5371b142eaf None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:13:01.354704 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 423/863] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:13:01 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:13:01.361353 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6981fbfa-1bf9-4e38-9a4f-c37c4e7c321d None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:13:01.455095 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6981fbfa-1bf9-4e38-9a4f-c37c4e7c321d None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:13:01.487558 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a69afca-052d-4b40-ae2b-b5f715e35949 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:13:01.488083 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a69afca-052d-4b40-ae2b-b5f715e35949 None admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:01.488352 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a69afca-052d-4b40-ae2b-b5f715e35949 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:01.488614 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a69afca-052d-4b40-ae2b-b5f715e35949 None admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:01.489270 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a69afca-052d-4b40-ae2b-b5f715e35949 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:13:01.490672 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a69afca-052d-4b40-ae2b-b5f715e35949 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:13:01.510920 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 441/864] 10.222.0.42 () {60 vars in 1132 bytes} [Thu Oct 27 01:13:01 2022] POST /identity/v3/users => generated 286 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:13:03.691249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0a1066-5209-4d9c-8696-19ef51eee538 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:03.691822 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0a1066-5209-4d9c-8696-19ef51eee538 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:03.691822 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0a1066-5209-4d9c-8696-19ef51eee538 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:03.693197 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 424/865] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:13:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:13:03.699133 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894159af-82fb-4780-b447-97b7ef0998bb None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:03.699576 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894159af-82fb-4780-b447-97b7ef0998bb None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:03.699668 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894159af-82fb-4780-b447-97b7ef0998bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:03.717346 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-894159af-82fb-4780-b447-97b7ef0998bb None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:13:03.718526 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-894159af-82fb-4780-b447-97b7ef0998bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:13:03.763299 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:13:03.784563 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 442/866] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:13:03 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:13:04.060390 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0a1066-5209-4d9c-8696-19ef51eee538 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:13:04.158968 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0a1066-5209-4d9c-8696-19ef51eee538 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:13:04.192180 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af6af7e-2544-4aee-93c9-9aa321153680 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:13:04.192788 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af6af7e-2544-4aee-93c9-9aa321153680 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:04.193091 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af6af7e-2544-4aee-93c9-9aa321153680 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:04.193360 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af6af7e-2544-4aee-93c9-9aa321153680 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:04.202971 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af6af7e-2544-4aee-93c9-9aa321153680 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:13:04.208127 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-8af6af7e-2544-4aee-93c9-9aa321153680 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:13:04.208127 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:13:04.208127 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:13:04.208127 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:13:04.208127 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:13:04.208127 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:13:04.208127 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:13:04.208127 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:13:04.208127 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:13:04.208127 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:13:04.208127 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:13:04.208127 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:13:04.208127 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:13:04.208127 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:13:04.208127 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:13:04.208127 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:13:04.209282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:13:04.209282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:13:04.209282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:13:04.209282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:13:04.209282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:13:04.209282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:13:04.209282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:13:04.209282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:13:04.209282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:13:04.209282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:13:04.209282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:13:04.209282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:13:04.209282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:13:04.209282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:13:04.209282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:13:04.209282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:13:04.210159 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:13:04.210159 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:13:04.210159 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:13:04.210159 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:13:04.210159 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:13:04.210159 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:13:04.210159 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:13:04.210159 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:13:04.210159 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:13:04.210159 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:13:04.210159 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:13:04.210159 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:13:04.210159 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 425/867] 10.222.0.42 () {56 vars in 1096 bytes} [Thu Oct 27 01:13:04 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:13:04.215140 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894159af-82fb-4780-b447-97b7ef0998bb None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:13:04.308897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894159af-82fb-4780-b447-97b7ef0998bb None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:13:04.341254 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818583f8-bfb2-4d76-8f82-803e3564e407 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:13:04.341782 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818583f8-bfb2-4d76-8f82-803e3564e407 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:04.342053 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818583f8-bfb2-4d76-8f82-803e3564e407 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:04.342320 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818583f8-bfb2-4d76-8f82-803e3564e407 None admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:04.342963 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-818583f8-bfb2-4d76-8f82-803e3564e407 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:13:04.343258 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-818583f8-bfb2-4d76-8f82-803e3564e407 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:13:04.344599 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-818583f8-bfb2-4d76-8f82-803e3564e407 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:13:04.352872 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 443/868] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:13:04 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:13:04.358767 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af6af7e-2544-4aee-93c9-9aa321153680 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:13:04.413704 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:13:04.455063 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af6af7e-2544-4aee-93c9-9aa321153680 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:13:04.487217 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6c1d37-54db-47d3-93a0-b2cf507db7db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:13:04.487827 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6c1d37-54db-47d3-93a0-b2cf507db7db None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:04.488099 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6c1d37-54db-47d3-93a0-b2cf507db7db None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:04.488364 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6c1d37-54db-47d3-93a0-b2cf507db7db None admin] PATH_INFO: `/v3/users/d1d8ebfe1ef44818b1faf4b8582ef519` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:04.500740 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af6c1d37-54db-47d3-93a0-b2cf507db7db None admin] RBAC: Authorizing `identity:get_user(user_id=d1d8ebfe1ef44818b1faf4b8582ef519)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:13:04.502703 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af6c1d37-54db-47d3-93a0-b2cf507db7db None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:13:04.504258 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 426/869] 10.222.0.42 () {56 vars in 1177 bytes} [Thu Oct 27 01:13:04 2022] GET /identity/v3/users/d1d8ebfe1ef44818b1faf4b8582ef519 => generated 286 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:13:04.510013 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818583f8-bfb2-4d76-8f82-803e3564e407 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:13:04.552127 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:13:04.607549 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818583f8-bfb2-4d76-8f82-803e3564e407 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:13:04.640479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb7f2b3-1eb1-4fbd-a6ea-1196a7840abf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:13:04.641119 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb7f2b3-1eb1-4fbd-a6ea-1196a7840abf None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:04.641401 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb7f2b3-1eb1-4fbd-a6ea-1196a7840abf None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:04.642231 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb7f2b3-1eb1-4fbd-a6ea-1196a7840abf None admin] PATH_INFO: `/v3/projects/06ab73e79cd04e63be6d0537145aac66` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:04.643339 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb7f2b3-1eb1-4fbd-a6ea-1196a7840abf None admin] RBAC: Authorizing `identity:get_project(project_id=06ab73e79cd04e63be6d0537145aac66)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:13:04.644752 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb7f2b3-1eb1-4fbd-a6ea-1196a7840abf None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:13:04.646375 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 444/870] 10.222.0.42 () {56 vars in 1186 bytes} [Thu Oct 27 01:13:04 2022] GET /identity/v3/projects/06ab73e79cd04e63be6d0537145aac66 => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:13:04.652234 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6c1d37-54db-47d3-93a0-b2cf507db7db None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:13:04.752541 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6c1d37-54db-47d3-93a0-b2cf507db7db None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:13:04.765747 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:13:04.786837 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d3619e-5e09-426e-96c2-ee69367e9133 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:13:04.787472 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d3619e-5e09-426e-96c2-ee69367e9133 None admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:04.787744 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d3619e-5e09-426e-96c2-ee69367e9133 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:04.788008 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d3619e-5e09-426e-96c2-ee69367e9133 None admin] PATH_INFO: `/v3/projects/06ab73e79cd04e63be6d0537145aac66/users/d1d8ebfe1ef44818b1faf4b8582ef519/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:04.789884 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7d3619e-5e09-426e-96c2-ee69367e9133 None admin] RBAC: Authorizing `identity:create_grant(project_id=06ab73e79cd04e63be6d0537145aac66, user_id=d1d8ebfe1ef44818b1faf4b8582ef519, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:13:04.791293 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7d3619e-5e09-426e-96c2-ee69367e9133 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:13:04.805627 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 427/871] 10.222.0.42 () {58 vars in 1439 bytes} [Thu Oct 27 01:13:04 2022] PUT /identity/v3/projects/06ab73e79cd04e63be6d0537145aac66/users/d1d8ebfe1ef44818b1faf4b8582ef519/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:13:06.975476 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa19a2d-5f6d-4566-be59-f53a7bc469a2 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:06.975996 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa19a2d-5f6d-4566-be59-f53a7bc469a2 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:06.975996 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa19a2d-5f6d-4566-be59-f53a7bc469a2 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:06.977375 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 445/872] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:13:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:13:06.982495 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c2f693-fe08-41fe-9007-32bf5fc35720 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:06.982912 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c2f693-fe08-41fe-9007-32bf5fc35720 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:06.983036 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c2f693-fe08-41fe-9007-32bf5fc35720 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:07.001522 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-89c2f693-fe08-41fe-9007-32bf5fc35720 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:13:07.002749 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-89c2f693-fe08-41fe-9007-32bf5fc35720 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:13:07.065729 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 428/873] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:13:06 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:13:07.335199 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa19a2d-5f6d-4566-be59-f53a7bc469a2 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:13:07.433450 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa19a2d-5f6d-4566-be59-f53a7bc469a2 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:13:07.443821 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:13:07.466797 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e563166-d33b-42c1-9f69-d14b6ce07f3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:13:07.467405 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e563166-d33b-42c1-9f69-d14b6ce07f3d None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:07.467681 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e563166-d33b-42c1-9f69-d14b6ce07f3d None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:07.467950 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e563166-d33b-42c1-9f69-d14b6ce07f3d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:07.479388 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e563166-d33b-42c1-9f69-d14b6ce07f3d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:13:07.484113 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-2e563166-d33b-42c1-9f69-d14b6ce07f3d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:13:07.484113 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:13:07.484113 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:13:07.484113 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:13:07.484113 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:13:07.484113 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:13:07.484113 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:13:07.484113 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:13:07.484113 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:13:07.484113 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:13:07.484113 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:13:07.484113 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:13:07.484113 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:13:07.484113 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:13:07.484113 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:13:07.484113 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:13:07.485719 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:13:07.485719 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:13:07.485719 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:13:07.485719 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:13:07.485719 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:13:07.485719 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:13:07.485719 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:13:07.485719 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:13:07.485719 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:13:07.485719 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:13:07.485719 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:13:07.485719 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:13:07.485719 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:13:07.485719 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:13:07.485719 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:13:07.485719 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:13:07.486775 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:13:07.486775 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:13:07.486775 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:13:07.486775 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:13:07.486775 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:13:07.486775 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:13:07.486775 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:13:07.486775 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:13:07.486775 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:13:07.486775 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:13:07.486775 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:13:07.486775 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:13:07.486775 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 446/874] 10.222.0.42 () {56 vars in 1096 bytes} [Thu Oct 27 01:13:07 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:13:07.491402 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c2f693-fe08-41fe-9007-32bf5fc35720 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:13:07.586265 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c2f693-fe08-41fe-9007-32bf5fc35720 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:13:07.588412 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:13:07.619072 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d576d984-f543-4fea-8546-d75f65f1f814 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:13:07.619662 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d576d984-f543-4fea-8546-d75f65f1f814 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:07.619943 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d576d984-f543-4fea-8546-d75f65f1f814 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:07.620212 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d576d984-f543-4fea-8546-d75f65f1f814 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:07.620866 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d576d984-f543-4fea-8546-d75f65f1f814 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:13:07.621192 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d576d984-f543-4fea-8546-d75f65f1f814 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:13:07.622551 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d576d984-f543-4fea-8546-d75f65f1f814 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:13:07.630881 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 429/875] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:13:07 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:13:07.636794 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e563166-d33b-42c1-9f69-d14b6ce07f3d None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:13:07.732346 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e563166-d33b-42c1-9f69-d14b6ce07f3d None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:13:07.764226 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d27785-9508-4c76-b8a6-bfd5e32fe2d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:13:07.764834 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d27785-9508-4c76-b8a6-bfd5e32fe2d8 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:07.765134 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d27785-9508-4c76-b8a6-bfd5e32fe2d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:07.765403 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d27785-9508-4c76-b8a6-bfd5e32fe2d8 None admin] PATH_INFO: `/v3/users/d1d8ebfe1ef44818b1faf4b8582ef519` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:07.766571 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d27785-9508-4c76-b8a6-bfd5e32fe2d8 None admin] RBAC: Authorizing `identity:get_user(user_id=d1d8ebfe1ef44818b1faf4b8582ef519)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:13:07.768169 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d27785-9508-4c76-b8a6-bfd5e32fe2d8 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:13:07.769738 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 447/876] 10.222.0.42 () {56 vars in 1177 bytes} [Thu Oct 27 01:13:07 2022] GET /identity/v3/users/d1d8ebfe1ef44818b1faf4b8582ef519 => generated 286 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:13:07.775664 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d576d984-f543-4fea-8546-d75f65f1f814 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:13:07.872587 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d576d984-f543-4fea-8546-d75f65f1f814 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:13:07.904295 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47659d4-52d2-4373-8036-c3be093461ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:13:07.904881 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47659d4-52d2-4373-8036-c3be093461ce None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:07.905180 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47659d4-52d2-4373-8036-c3be093461ce None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:07.905449 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47659d4-52d2-4373-8036-c3be093461ce None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:07.936781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:13:07.969895 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c47659d4-52d2-4373-8036-c3be093461ce None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:13:07.971309 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c47659d4-52d2-4373-8036-c3be093461ce None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:13:08.063127 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 430/877] 10.222.0.42 () {58 vars in 1282 bytes} [Thu Oct 27 01:13:07 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 288 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 27 01:13:08.069170 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d27785-9508-4c76-b8a6-bfd5e32fe2d8 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:13:08.109471 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:13:08.167403 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d27785-9508-4c76-b8a6-bfd5e32fe2d8 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:13:08.199700 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d2f865-aefc-46d1-a067-ebe45ca7d309 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:13:08.200311 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d2f865-aefc-46d1-a067-ebe45ca7d309 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:08.200587 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d2f865-aefc-46d1-a067-ebe45ca7d309 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:08.200854 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d2f865-aefc-46d1-a067-ebe45ca7d309 None admin] PATH_INFO: `/v3/projects/06ab73e79cd04e63be6d0537145aac66` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:08.201977 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79d2f865-aefc-46d1-a067-ebe45ca7d309 None admin] RBAC: Authorizing `identity:get_project(project_id=06ab73e79cd04e63be6d0537145aac66)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:13:08.203396 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79d2f865-aefc-46d1-a067-ebe45ca7d309 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:13:08.205534 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 448/878] 10.222.0.42 () {56 vars in 1186 bytes} [Thu Oct 27 01:13:08 2022] GET /identity/v3/projects/06ab73e79cd04e63be6d0537145aac66 => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:13:08.211629 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47659d4-52d2-4373-8036-c3be093461ce None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:13:08.274845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:13:08.307807 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47659d4-52d2-4373-8036-c3be093461ce None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:13:08.340725 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691657be-12aa-4a87-8ccc-2bbc356698a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:13:08.341336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691657be-12aa-4a87-8ccc-2bbc356698a0 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:08.341611 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691657be-12aa-4a87-8ccc-2bbc356698a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:08.341878 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691657be-12aa-4a87-8ccc-2bbc356698a0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:08.342593 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691657be-12aa-4a87-8ccc-2bbc356698a0 None admin] RBAC: Adding query filter params (role.id=414286927fdd44688107ff5e10235acf, scope.project.id=06ab73e79cd04e63be6d0537145aac66, user.id=d1d8ebfe1ef44818b1faf4b8582ef519) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:13:08.342896 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691657be-12aa-4a87-8ccc-2bbc356698a0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:13:08.344307 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691657be-12aa-4a87-8ccc-2bbc356698a0 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:13:08.349848 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 431/879] 10.222.0.42 () {56 vars in 1374 bytes} [Thu Oct 27 01:13:08 2022] GET /identity/v3/role_assignments?user.id=d1d8ebfe1ef44818b1faf4b8582ef519&scope.project.id=06ab73e79cd04e63be6d0537145aac66&role.id=414286927fdd44688107ff5e10235acf => generated 608 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:13:10.487562 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187aa65c-3fc8-4957-b602-fa53817385b9 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:10.488091 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187aa65c-3fc8-4957-b602-fa53817385b9 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:10.488091 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187aa65c-3fc8-4957-b602-fa53817385b9 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:10.489439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 449/880] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:13:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:13:10.494589 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea147c15-ad96-4d96-9f21-1ae4803509f6 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:10.494977 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea147c15-ad96-4d96-9f21-1ae4803509f6 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:10.495120 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea147c15-ad96-4d96-9f21-1ae4803509f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:10.513591 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-ea147c15-ad96-4d96-9f21-1ae4803509f6 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:13:10.514746 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ea147c15-ad96-4d96-9f21-1ae4803509f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:13:10.580011 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 432/881] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:13:10 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:13:10.854230 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-187aa65c-3fc8-4957-b602-fa53817385b9 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:13:10.952687 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-187aa65c-3fc8-4957-b602-fa53817385b9 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:13:10.959807 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:13:10.986989 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d8c075-49f9-4fd5-9f1c-671e2a783a42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:13:10.987573 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d8c075-49f9-4fd5-9f1c-671e2a783a42 None admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:10.987849 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d8c075-49f9-4fd5-9f1c-671e2a783a42 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:10.988118 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d8c075-49f9-4fd5-9f1c-671e2a783a42 None admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:10.988777 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d8c075-49f9-4fd5-9f1c-671e2a783a42 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:13:10.990162 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d8c075-49f9-4fd5-9f1c-671e2a783a42 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:13:11.006190 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 450/882] 10.222.0.42 () {60 vars in 1138 bytes} [Thu Oct 27 01:13:10 2022] POST /identity/v3/domains => generated 267 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:13:13.175726 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47494b2a-2620-4f21-a14a-1ad792fc4a6d None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:13.176296 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47494b2a-2620-4f21-a14a-1ad792fc4a6d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:13.176296 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47494b2a-2620-4f21-a14a-1ad792fc4a6d None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:13.177667 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 433/883] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:13:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:13:13.183020 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ed9381-1e26-40bb-9c67-5e24298c4d33 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:13.183409 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ed9381-1e26-40bb-9c67-5e24298c4d33 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:13.183566 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ed9381-1e26-40bb-9c67-5e24298c4d33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:13.201762 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-88ed9381-1e26-40bb-9c67-5e24298c4d33 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:13:13.202932 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-88ed9381-1e26-40bb-9c67-5e24298c4d33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:13:13.267014 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 451/884] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:13:13 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:13:13.541742 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47494b2a-2620-4f21-a14a-1ad792fc4a6d None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:13:13.601042 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:13:13.641046 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47494b2a-2620-4f21-a14a-1ad792fc4a6d None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:13:13.673892 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26662dec-4715-4a12-b5ad-de27276d96ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:13:13.674499 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26662dec-4715-4a12-b5ad-de27276d96ae None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:13.674774 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26662dec-4715-4a12-b5ad-de27276d96ae None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:13.675040 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26662dec-4715-4a12-b5ad-de27276d96ae None admin] PATH_INFO: `/v3/domains/a54e276161944f6da56da24f2f5f2ba6` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:13.685267 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26662dec-4715-4a12-b5ad-de27276d96ae None admin] RBAC: Authorizing `identity:get_domain(domain_id=a54e276161944f6da56da24f2f5f2ba6)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:13:13.686697 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26662dec-4715-4a12-b5ad-de27276d96ae None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:13:13.688797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 434/885] 10.222.0.42 () {56 vars in 1183 bytes} [Thu Oct 27 01:13:13 2022] GET /identity/v3/domains/a54e276161944f6da56da24f2f5f2ba6 => generated 267 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:13:13.694609 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ed9381-1e26-40bb-9c67-5e24298c4d33 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:13:13.784570 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:13:13.790686 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ed9381-1e26-40bb-9c67-5e24298c4d33 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:13:13.825130 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a955fa-a869-4eee-b873-73216e5f4c60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:13:13.825729 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a955fa-a869-4eee-b873-73216e5f4c60 None admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:13.826003 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a955fa-a869-4eee-b873-73216e5f4c60 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:13.826269 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a955fa-a869-4eee-b873-73216e5f4c60 None admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:13.826908 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a955fa-a869-4eee-b873-73216e5f4c60 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:13:13.828323 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a955fa-a869-4eee-b873-73216e5f4c60 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:13:13.854701 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 452/886] 10.222.0.42 () {60 vars in 1141 bytes} [Thu Oct 27 01:13:13 2022] POST /identity/v3/projects => generated 361 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:13:16.064239 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929c82f4-52c0-4237-a48e-4a0137693816 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:16.064762 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929c82f4-52c0-4237-a48e-4a0137693816 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:16.064762 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929c82f4-52c0-4237-a48e-4a0137693816 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:16.066067 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 435/887] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:13:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:13:16.071259 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967cebaa-4967-489d-a1f9-40e5764a5d6c None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:16.071688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967cebaa-4967-489d-a1f9-40e5764a5d6c None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:16.071838 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967cebaa-4967-489d-a1f9-40e5764a5d6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:16.090042 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-967cebaa-4967-489d-a1f9-40e5764a5d6c None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:13:16.091211 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-967cebaa-4967-489d-a1f9-40e5764a5d6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:13:16.155182 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 453/888] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:13:16 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 27 01:13:16.426460 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-929c82f4-52c0-4237-a48e-4a0137693816 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:13:16.523442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-929c82f4-52c0-4237-a48e-4a0137693816 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:13:16.543225 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:13:16.557731 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2829a382-c7a3-4069-88a9-af4f1c56a7c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:13:16.558342 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2829a382-c7a3-4069-88a9-af4f1c56a7c6 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:16.558617 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2829a382-c7a3-4069-88a9-af4f1c56a7c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:16.558884 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2829a382-c7a3-4069-88a9-af4f1c56a7c6 None admin] PATH_INFO: `/v3/domains/a54e276161944f6da56da24f2f5f2ba6` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:16.559975 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2829a382-c7a3-4069-88a9-af4f1c56a7c6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=a54e276161944f6da56da24f2f5f2ba6)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:13:16.561417 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2829a382-c7a3-4069-88a9-af4f1c56a7c6 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:13:16.562968 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 436/889] 10.222.0.42 () {56 vars in 1183 bytes} [Thu Oct 27 01:13:16 2022] GET /identity/v3/domains/a54e276161944f6da56da24f2f5f2ba6 => generated 267 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:13:16.568679 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967cebaa-4967-489d-a1f9-40e5764a5d6c None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:13:16.643066 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:13:16.664950 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967cebaa-4967-489d-a1f9-40e5764a5d6c None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:13:16.698606 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4bce12-8f84-4880-b900-27b124ba364f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:13:16.699137 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4bce12-8f84-4880-b900-27b124ba364f None admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:16.699413 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4bce12-8f84-4880-b900-27b124ba364f None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:16.699679 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4bce12-8f84-4880-b900-27b124ba364f None admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:16.700318 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab4bce12-8f84-4880-b900-27b124ba364f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:13:16.701760 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab4bce12-8f84-4880-b900-27b124ba364f None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:13:16.722303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 454/890] 10.222.0.42 () {60 vars in 1132 bytes} [Thu Oct 27 01:13:16 2022] POST /identity/v3/users => generated 311 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:13:18.902368 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9f4a1e-d849-4fae-b907-3e1f96aaf1e6 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:18.902874 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9f4a1e-d849-4fae-b907-3e1f96aaf1e6 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:18.902874 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9f4a1e-d849-4fae-b907-3e1f96aaf1e6 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:18.904204 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 437/891] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:13:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:13:18.911481 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04190ea-7c2c-44e1-aa92-7faebc2da804 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:18.911880 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04190ea-7c2c-44e1-aa92-7faebc2da804 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:18.912030 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04190ea-7c2c-44e1-aa92-7faebc2da804 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:18.929837 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-a04190ea-7c2c-44e1-aa92-7faebc2da804 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:13:18.931270 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-a04190ea-7c2c-44e1-aa92-7faebc2da804 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:13:18.995461 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 455/892] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:13:18 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:13:19.267154 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9f4a1e-d849-4fae-b907-3e1f96aaf1e6 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:13:19.365793 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9f4a1e-d849-4fae-b907-3e1f96aaf1e6 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:13:19.398877 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2df1d9d-f20f-4642-87ea-88ff8eb79130 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:13:19.399479 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2df1d9d-f20f-4642-87ea-88ff8eb79130 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:19.399748 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2df1d9d-f20f-4642-87ea-88ff8eb79130 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:19.400012 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2df1d9d-f20f-4642-87ea-88ff8eb79130 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:19.409811 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2df1d9d-f20f-4642-87ea-88ff8eb79130 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:13:19.414488 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-f2df1d9d-f20f-4642-87ea-88ff8eb79130 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:13:19.414488 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:13:19.414488 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:13:19.414488 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:13:19.414488 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:13:19.414488 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:13:19.414488 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:13:19.414488 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:13:19.414488 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:13:19.414488 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:13:19.414488 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:13:19.414488 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:13:19.414488 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:13:19.414488 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:13:19.414488 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:13:19.414488 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:13:19.417152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:13:19.417152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:13:19.417152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:13:19.417152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:13:19.417152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:13:19.417152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:13:19.417152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:13:19.417152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:13:19.417152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:13:19.417152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:13:19.417152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:13:19.417152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:13:19.417152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:13:19.417152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:13:19.417152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:13:19.417152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:13:19.418345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:13:19.418345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:13:19.418345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:13:19.418345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:13:19.418345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:13:19.418345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:13:19.418345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:13:19.418345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:13:19.418345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:13:19.418345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:13:19.418345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:13:19.418345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:13:19.418345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 438/893] 10.222.0.42 () {56 vars in 1096 bytes} [Thu Oct 27 01:13:19 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:13:19.421684 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04190ea-7c2c-44e1-aa92-7faebc2da804 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:13:19.481851 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:13:19.519629 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04190ea-7c2c-44e1-aa92-7faebc2da804 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:13:19.553189 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b0cca6-614c-4c20-b070-36190da58664 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:13:19.553785 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b0cca6-614c-4c20-b070-36190da58664 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:19.554063 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b0cca6-614c-4c20-b070-36190da58664 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:19.554334 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b0cca6-614c-4c20-b070-36190da58664 None admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:19.554989 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48b0cca6-614c-4c20-b070-36190da58664 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:13:19.555292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48b0cca6-614c-4c20-b070-36190da58664 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:13:19.556901 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48b0cca6-614c-4c20-b070-36190da58664 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:13:19.565209 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 456/894] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:13:19 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:13:19.570972 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2df1d9d-f20f-4642-87ea-88ff8eb79130 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:13:19.619191 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:13:19.668126 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2df1d9d-f20f-4642-87ea-88ff8eb79130 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:13:19.700619 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f539721d-66fc-41f2-8d38-a61e2c0a1647 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:13:19.701257 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f539721d-66fc-41f2-8d38-a61e2c0a1647 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:19.701536 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f539721d-66fc-41f2-8d38-a61e2c0a1647 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:19.701805 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f539721d-66fc-41f2-8d38-a61e2c0a1647 None admin] PATH_INFO: `/v3/users/1fd2c11698ce4e5ea8ddb20b0e388640` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:19.714917 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f539721d-66fc-41f2-8d38-a61e2c0a1647 None admin] RBAC: Authorizing `identity:get_user(user_id=1fd2c11698ce4e5ea8ddb20b0e388640)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:13:19.716345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f539721d-66fc-41f2-8d38-a61e2c0a1647 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:13:19.717948 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 439/895] 10.222.0.42 () {56 vars in 1177 bytes} [Thu Oct 27 01:13:19 2022] GET /identity/v3/users/1fd2c11698ce4e5ea8ddb20b0e388640 => generated 311 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:13:19.723629 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b0cca6-614c-4c20-b070-36190da58664 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:13:19.820159 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b0cca6-614c-4c20-b070-36190da58664 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:13:19.844991 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:13:19.854185 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4bebb6-ac1e-4a24-b20b-e284a08107c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:13:19.854804 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4bebb6-ac1e-4a24-b20b-e284a08107c1 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:19.855081 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4bebb6-ac1e-4a24-b20b-e284a08107c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:19.855352 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4bebb6-ac1e-4a24-b20b-e284a08107c1 None admin] PATH_INFO: `/v3/projects/02681ee0df6e430781140feb44036cc3` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:19.856460 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4bebb6-ac1e-4a24-b20b-e284a08107c1 None admin] RBAC: Authorizing `identity:get_project(project_id=02681ee0df6e430781140feb44036cc3)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:13:19.857901 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4bebb6-ac1e-4a24-b20b-e284a08107c1 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:13:19.859498 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 457/896] 10.222.0.42 () {56 vars in 1186 bytes} [Thu Oct 27 01:13:19 2022] GET /identity/v3/projects/02681ee0df6e430781140feb44036cc3 => generated 361 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:13:19.865979 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f539721d-66fc-41f2-8d38-a61e2c0a1647 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:13:19.961385 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f539721d-66fc-41f2-8d38-a61e2c0a1647 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:13:19.968263 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:13:19.994445 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75cf853e-1dc8-47cc-b868-57ed574c413a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:13:19.995083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75cf853e-1dc8-47cc-b868-57ed574c413a None admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:19.995364 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75cf853e-1dc8-47cc-b868-57ed574c413a None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:19.995631 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75cf853e-1dc8-47cc-b868-57ed574c413a None admin] PATH_INFO: `/v3/projects/02681ee0df6e430781140feb44036cc3/users/1fd2c11698ce4e5ea8ddb20b0e388640/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:19.997512 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75cf853e-1dc8-47cc-b868-57ed574c413a None admin] RBAC: Authorizing `identity:create_grant(project_id=02681ee0df6e430781140feb44036cc3, user_id=1fd2c11698ce4e5ea8ddb20b0e388640, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:13:19.998925 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75cf853e-1dc8-47cc-b868-57ed574c413a None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:13:20.012374 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 440/897] 10.222.0.42 () {58 vars in 1439 bytes} [Thu Oct 27 01:13:19 2022] PUT /identity/v3/projects/02681ee0df6e430781140feb44036cc3/users/1fd2c11698ce4e5ea8ddb20b0e388640/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:13:22.138982 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145c8fcd-8978-495c-8072-9bae5bd960a2 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:22.139502 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145c8fcd-8978-495c-8072-9bae5bd960a2 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:22.139502 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145c8fcd-8978-495c-8072-9bae5bd960a2 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:22.141290 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 458/898] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:13:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:13:22.146708 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d841229-48e0-428f-9cd2-8281e80ce1d2 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:22.147083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d841229-48e0-428f-9cd2-8281e80ce1d2 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:22.147255 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d841229-48e0-428f-9cd2-8281e80ce1d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:22.165663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-9d841229-48e0-428f-9cd2-8281e80ce1d2 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:13:22.166847 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-9d841229-48e0-428f-9cd2-8281e80ce1d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:13:22.230388 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 441/899] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:13:22 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:13:22.500693 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-145c8fcd-8978-495c-8072-9bae5bd960a2 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:13:22.598500 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-145c8fcd-8978-495c-8072-9bae5bd960a2 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:13:22.630750 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce878fd7-cca5-42e3-9bfa-300809e403ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:13:22.631361 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce878fd7-cca5-42e3-9bfa-300809e403ae None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:22.631636 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce878fd7-cca5-42e3-9bfa-300809e403ae None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:22.631904 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce878fd7-cca5-42e3-9bfa-300809e403ae None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:22.642021 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce878fd7-cca5-42e3-9bfa-300809e403ae None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:13:22.646688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-ce878fd7-cca5-42e3-9bfa-300809e403ae None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:13:22.646688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:13:22.646688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:13:22.646688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:13:22.646688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:13:22.646688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:13:22.646688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:13:22.646688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:13:22.646688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:13:22.646688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:13:22.646688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:13:22.646688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:13:22.646688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:13:22.646688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:13:22.646688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:13:22.646688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:13:22.647767 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:13:22.647767 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:13:22.647767 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:13:22.647767 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:13:22.647767 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:13:22.647767 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:13:22.647767 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:13:22.647767 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:13:22.647767 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:13:22.647767 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:13:22.647767 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:13:22.647767 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:13:22.647767 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:13:22.647767 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:13:22.647767 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:13:22.647767 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:13:22.648856 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:13:22.648856 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:13:22.648856 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:13:22.648856 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:13:22.648856 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:13:22.648856 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:13:22.648856 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:13:22.648856 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:13:22.648856 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:13:22.648856 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:13:22.648856 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:13:22.648856 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:13:22.648856 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 459/900] 10.222.0.42 () {56 vars in 1096 bytes} [Thu Oct 27 01:13:22 2022] GET /identity/v3/roles/admin => generated 83 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:13:22.653693 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d841229-48e0-428f-9cd2-8281e80ce1d2 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:13:22.742776 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:13:22.748966 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d841229-48e0-428f-9cd2-8281e80ce1d2 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:13:22.782088 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c0887e-ad0b-4b22-8625-2d25383c9527 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:13:22.782711 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c0887e-ad0b-4b22-8625-2d25383c9527 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:22.782988 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c0887e-ad0b-4b22-8625-2d25383c9527 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:22.783261 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c0887e-ad0b-4b22-8625-2d25383c9527 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:22.783927 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c0887e-ad0b-4b22-8625-2d25383c9527 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:13:22.784227 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c0887e-ad0b-4b22-8625-2d25383c9527 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:13:22.785617 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c0887e-ad0b-4b22-8625-2d25383c9527 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:13:22.793263 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 442/901] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:13:22 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:13:22.799734 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce878fd7-cca5-42e3-9bfa-300809e403ae None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:13:22.854111 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:13:22.897179 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce878fd7-cca5-42e3-9bfa-300809e403ae None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:13:22.929643 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01976239-b278-4e87-a5b8-7ef90d51b42e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:13:22.930252 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01976239-b278-4e87-a5b8-7ef90d51b42e None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:22.930777 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01976239-b278-4e87-a5b8-7ef90d51b42e None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:22.931043 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01976239-b278-4e87-a5b8-7ef90d51b42e None admin] PATH_INFO: `/v3/users/1fd2c11698ce4e5ea8ddb20b0e388640` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:22.932132 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01976239-b278-4e87-a5b8-7ef90d51b42e None admin] RBAC: Authorizing `identity:get_user(user_id=1fd2c11698ce4e5ea8ddb20b0e388640)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:13:22.933564 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01976239-b278-4e87-a5b8-7ef90d51b42e None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:13:22.935097 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 460/902] 10.222.0.42 () {56 vars in 1177 bytes} [Thu Oct 27 01:13:22 2022] GET /identity/v3/users/1fd2c11698ce4e5ea8ddb20b0e388640 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:13:22.940919 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c0887e-ad0b-4b22-8625-2d25383c9527 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:13:23.036047 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c0887e-ad0b-4b22-8625-2d25383c9527 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:13:23.067582 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc078bb-4a5c-42fd-952c-d3a738475572 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:13:23.068195 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc078bb-4a5c-42fd-952c-d3a738475572 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:23.068474 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc078bb-4a5c-42fd-952c-d3a738475572 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:23.068745 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc078bb-4a5c-42fd-952c-d3a738475572 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:23.095323 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:13:23.134097 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfc078bb-4a5c-42fd-952c-d3a738475572 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:13:23.135516 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfc078bb-4a5c-42fd-952c-d3a738475572 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:13:23.229651 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 443/903] 10.222.0.42 () {58 vars in 1282 bytes} [Thu Oct 27 01:13:22 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 289 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 27 01:13:23.235650 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01976239-b278-4e87-a5b8-7ef90d51b42e None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:13:23.333140 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01976239-b278-4e87-a5b8-7ef90d51b42e None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:13:23.354071 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:13:23.365747 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a7b386-0471-421c-a62f-d8a68b3ffe84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:13:23.366357 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a7b386-0471-421c-a62f-d8a68b3ffe84 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:23.366631 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a7b386-0471-421c-a62f-d8a68b3ffe84 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:23.366898 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a7b386-0471-421c-a62f-d8a68b3ffe84 None admin] PATH_INFO: `/v3/projects/02681ee0df6e430781140feb44036cc3` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:23.367993 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a7b386-0471-421c-a62f-d8a68b3ffe84 None admin] RBAC: Authorizing `identity:get_project(project_id=02681ee0df6e430781140feb44036cc3)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:13:23.369449 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a7b386-0471-421c-a62f-d8a68b3ffe84 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:13:23.371021 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 461/904] 10.222.0.42 () {56 vars in 1186 bytes} [Thu Oct 27 01:13:23 2022] GET /identity/v3/projects/02681ee0df6e430781140feb44036cc3 => generated 361 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:13:23.377780 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc078bb-4a5c-42fd-952c-d3a738475572 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:13:23.434301 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:13:23.473756 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc078bb-4a5c-42fd-952c-d3a738475572 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:13:23.506405 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1414668f-09aa-4730-9738-0ab4f33e9766 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:13:23.506991 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1414668f-09aa-4730-9738-0ab4f33e9766 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:13:23.507266 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1414668f-09aa-4730-9738-0ab4f33e9766 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:13:23.507534 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1414668f-09aa-4730-9738-0ab4f33e9766 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:13:23.508243 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1414668f-09aa-4730-9738-0ab4f33e9766 None admin] RBAC: Adding query filter params (role.id=414286927fdd44688107ff5e10235acf, scope.project.id=02681ee0df6e430781140feb44036cc3, user.id=1fd2c11698ce4e5ea8ddb20b0e388640) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:13:23.508553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1414668f-09aa-4730-9738-0ab4f33e9766 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:13:23.509984 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1414668f-09aa-4730-9738-0ab4f33e9766 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:13:23.515450 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 444/905] 10.222.0.42 () {56 vars in 1374 bytes} [Thu Oct 27 01:13:23 2022] GET /identity/v3/role_assignments?user.id=1fd2c11698ce4e5ea8ddb20b0e388640&scope.project.id=02681ee0df6e430781140feb44036cc3&role.id=414286927fdd44688107ff5e10235acf => generated 608 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:13:33.361731 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=5 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:13:33.513392 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=39 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:19.511734 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9eb6b9d-119d-44a8-a2b7-9bdc1950ac79 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:19.512334 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9eb6b9d-119d-44a8-a2b7-9bdc1950ac79 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:19.512334 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9eb6b9d-119d-44a8-a2b7-9bdc1950ac79 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:19.513963 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 462/906] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:14:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 27 01:14:19.520379 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fef21af-d63c-4d50-8334-d651825c7b09 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:19.520877 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fef21af-d63c-4d50-8334-d651825c7b09 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:19.520969 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fef21af-d63c-4d50-8334-d651825c7b09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:19.540697 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-4fef21af-d63c-4d50-8334-d651825c7b09 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:14:19.542045 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-4fef21af-d63c-4d50-8334-d651825c7b09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:14:19.634710 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 445/907] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:14:19 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:14:19.922550 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9eb6b9d-119d-44a8-a2b7-9bdc1950ac79 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:20.049798 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9eb6b9d-119d-44a8-a2b7-9bdc1950ac79 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:20.092465 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852581ab-8898-4e62-8440-bc22b0aa20b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:20.093026 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852581ab-8898-4e62-8440-bc22b0aa20b3 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:20.093457 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852581ab-8898-4e62-8440-bc22b0aa20b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:20.093724 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852581ab-8898-4e62-8440-bc22b0aa20b3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:20.106056 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852581ab-8898-4e62-8440-bc22b0aa20b3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:20.109111 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852581ab-8898-4e62-8440-bc22b0aa20b3 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:20.125748 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-852581ab-8898-4e62-8440-bc22b0aa20b3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:14:20.125748 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:14:20.125748 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:14:20.125748 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:14:20.125748 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:14:20.125748 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:14:20.125748 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:14:20.125748 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:14:20.125748 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:14:20.125748 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:14:20.125748 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:14:20.125748 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:14:20.125748 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:14:20.125748 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:14:20.125748 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:14:20.125748 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:14:20.126876 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:14:20.126876 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:14:20.126876 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:14:20.126876 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:14:20.126876 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:14:20.126876 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:14:20.126876 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:14:20.126876 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:14:20.126876 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:14:20.126876 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:14:20.126876 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:14:20.126876 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:14:20.126876 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:14:20.126876 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:14:20.126876 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:14:20.126876 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:14:20.126876 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:14:20.127841 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:14:20.127841 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:14:20.127841 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:14:20.127841 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 463/908] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:14:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 27 01:14:20.133642 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fef21af-d63c-4d50-8334-d651825c7b09 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:20.253505 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fef21af-d63c-4d50-8334-d651825c7b09 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:20.285111 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:20.292386 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f48c66-5d83-4ed7-927c-caf05d2e3809 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:20.293188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f48c66-5d83-4ed7-927c-caf05d2e3809 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:20.293639 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f48c66-5d83-4ed7-927c-caf05d2e3809 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:20.294003 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f48c66-5d83-4ed7-927c-caf05d2e3809 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:20.294984 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f48c66-5d83-4ed7-927c-caf05d2e3809 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:14:20.295288 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f48c66-5d83-4ed7-927c-caf05d2e3809 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:20.296645 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f48c66-5d83-4ed7-927c-caf05d2e3809 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:20.309446 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 446/909] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:14:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:14:20.315591 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852581ab-8898-4e62-8440-bc22b0aa20b3 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:20.376837 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:20.443358 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852581ab-8898-4e62-8440-bc22b0aa20b3 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:20.489712 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63864741-a905-40ff-92e3-d426158d0471 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:20.490350 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63864741-a905-40ff-92e3-d426158d0471 None admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:20.490638 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63864741-a905-40ff-92e3-d426158d0471 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:20.490896 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63864741-a905-40ff-92e3-d426158d0471 None admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:20.491534 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63864741-a905-40ff-92e3-d426158d0471 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:20.493165 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63864741-a905-40ff-92e3-d426158d0471 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:20.523474 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 464/910] 10.222.0.42 () {60 vars in 1132 bytes} [Thu Oct 27 01:14:20 2022] POST /identity/v3/users => generated 251 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:14:22.875555 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb902a8-66ad-4920-af0c-372777311c60 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:22.876087 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb902a8-66ad-4920-af0c-372777311c60 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:22.876087 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb902a8-66ad-4920-af0c-372777311c60 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:22.877460 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 447/911] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:14:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:14:22.883690 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52871ddf-f852-43f8-b300-ab5fa14ea6a0 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:22.884193 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52871ddf-f852-43f8-b300-ab5fa14ea6a0 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:22.884287 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52871ddf-f852-43f8-b300-ab5fa14ea6a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:22.906324 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-52871ddf-f852-43f8-b300-ab5fa14ea6a0 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:14:22.907950 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-52871ddf-f852-43f8-b300-ab5fa14ea6a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:14:22.997101 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 465/912] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:14:22 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:14:23.296232 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb902a8-66ad-4920-af0c-372777311c60 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:23.433049 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb902a8-66ad-4920-af0c-372777311c60 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:23.483213 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6475e573-ede5-4db3-869a-fb89c26352de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:23.483983 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6475e573-ede5-4db3-869a-fb89c26352de None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:23.484280 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6475e573-ede5-4db3-869a-fb89c26352de None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:23.484618 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6475e573-ede5-4db3-869a-fb89c26352de None admin] PATH_INFO: `/v3/roles/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:23.496860 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6475e573-ede5-4db3-869a-fb89c26352de None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:14:23.502441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-6475e573-ede5-4db3-869a-fb89c26352de None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 27 01:14:23.502441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:14:23.502441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:14:23.502441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:14:23.502441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:14:23.502441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:14:23.502441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:14:23.502441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:14:23.502441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:14:23.502441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:14:23.502441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:14:23.502441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:14:23.502441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:14:23.502441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:14:23.502441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:14:23.502441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:14:23.506349 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:14:23.506349 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:14:23.506349 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:14:23.506349 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:14:23.506349 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:14:23.506349 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:14:23.506349 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:14:23.506349 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:14:23.506349 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:14:23.506349 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:14:23.506349 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:14:23.506349 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:14:23.506349 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:14:23.506349 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:14:23.506349 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:14:23.506349 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:14:23.507219 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:14:23.507219 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:14:23.507219 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:14:23.507219 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:14:23.507219 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:14:23.507219 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:14:23.507219 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:14:23.507219 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:14:23.507219 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:14:23.507219 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:14:23.507219 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 27 01:14:23.507219 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:14:23.507219 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 448/913] 10.222.0.42 () {56 vars in 1102 bytes} [Thu Oct 27 01:14:23 2022] GET /identity/v3/roles/service => generated 85 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:14:23.510803 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52871ddf-f852-43f8-b300-ab5fa14ea6a0 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:23.566371 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:23.643956 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52871ddf-f852-43f8-b300-ab5fa14ea6a0 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:23.682923 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7589238b-bf19-4ba0-b486-e27e758dcffa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:23.684367 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7589238b-bf19-4ba0-b486-e27e758dcffa None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:23.684367 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7589238b-bf19-4ba0-b486-e27e758dcffa None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:23.684367 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7589238b-bf19-4ba0-b486-e27e758dcffa None admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:23.684990 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7589238b-bf19-4ba0-b486-e27e758dcffa None admin] RBAC: Adding query filter params (name=service) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:14:23.685660 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7589238b-bf19-4ba0-b486-e27e758dcffa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:23.687060 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7589238b-bf19-4ba0-b486-e27e758dcffa None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:23.701856 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 466/914] 10.222.0.42 () {56 vars in 1103 bytes} [Thu Oct 27 01:14:23 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:14:23.704288 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6475e573-ede5-4db3-869a-fb89c26352de None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:23.764213 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:23.828437 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6475e573-ede5-4db3-869a-fb89c26352de None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:23.868981 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebff36fe-7b31-4eb6-86fe-be0ab5392191 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:23.869648 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebff36fe-7b31-4eb6-86fe-be0ab5392191 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:23.869925 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebff36fe-7b31-4eb6-86fe-be0ab5392191 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:23.870194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebff36fe-7b31-4eb6-86fe-be0ab5392191 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:23.882495 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebff36fe-7b31-4eb6-86fe-be0ab5392191 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:23.884378 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebff36fe-7b31-4eb6-86fe-be0ab5392191 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:23.902781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-ebff36fe-7b31-4eb6-86fe-be0ab5392191 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:14:23.902781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:14:23.902781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:14:23.902781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:14:23.902781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:14:23.902781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:14:23.902781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:14:23.902781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:14:23.902781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:14:23.902781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:14:23.902781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:14:23.902781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:14:23.902781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:14:23.902781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:14:23.902781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:14:23.902781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:14:23.904310 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:14:23.904310 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:14:23.904310 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:14:23.904310 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:14:23.904310 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:14:23.904310 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:14:23.904310 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:14:23.904310 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:14:23.904310 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:14:23.904310 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:14:23.904310 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:14:23.904310 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:14:23.904310 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:14:23.904310 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:14:23.904310 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:14:23.904310 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:14:23.904310 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:14:23.905238 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:14:23.905238 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:14:23.905238 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:14:23.910263 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 449/915] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:14:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:14:23.912110 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7589238b-bf19-4ba0-b486-e27e758dcffa None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:24.030047 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:24.033498 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7589238b-bf19-4ba0-b486-e27e758dcffa None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:24.079001 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e6fa80-be0b-4650-aa86-db19aa7da4c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:24.079530 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e6fa80-be0b-4650-aa86-db19aa7da4c6 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:24.079781 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e6fa80-be0b-4650-aa86-db19aa7da4c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:24.080049 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e6fa80-be0b-4650-aa86-db19aa7da4c6 None admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:24.080688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e6fa80-be0b-4650-aa86-db19aa7da4c6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:14:24.081017 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e6fa80-be0b-4650-aa86-db19aa7da4c6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:24.082379 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e6fa80-be0b-4650-aa86-db19aa7da4c6 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:24.096673 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 467/916] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:14:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:14:24.103640 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebff36fe-7b31-4eb6-86fe-be0ab5392191 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:24.225996 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebff36fe-7b31-4eb6-86fe-be0ab5392191 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:24.227538 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:24.264245 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2069a339-c390-4632-939c-118e39deff32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:24.264886 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2069a339-c390-4632-939c-118e39deff32 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:24.265180 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2069a339-c390-4632-939c-118e39deff32 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:24.265450 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2069a339-c390-4632-939c-118e39deff32 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:24.273540 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2069a339-c390-4632-939c-118e39deff32 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:24.275002 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2069a339-c390-4632-939c-118e39deff32 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:24.286550 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-2069a339-c390-4632-939c-118e39deff32 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 27 01:14:24.286550 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:14:24.286550 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:14:24.286550 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:14:24.286550 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:14:24.286550 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:14:24.286550 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:14:24.286550 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:14:24.286550 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:14:24.286550 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:14:24.286550 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:14:24.286550 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:14:24.286550 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 27 01:14:24.286550 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 27 01:14:24.286550 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 27 01:14:24.286550 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 27 01:14:24.287794 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:14:24.287794 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:14:24.287794 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 27 01:14:24.287794 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:14:24.287794 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 27 01:14:24.287794 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:14:24.287794 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:14:24.287794 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:14:24.287794 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:14:24.287794 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:14:24.287794 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:14:24.287794 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:14:24.287794 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:14:24.287794 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:14:24.287794 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:14:24.287794 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:14:24.287794 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:14:24.287794 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:14:24.288970 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:14:24.288970 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:14:24.288970 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 27 01:14:24.288970 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 27 01:14:24.288970 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 27 01:14:24.288970 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 27 01:14:24.288970 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 27 01:14:24.288970 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 27 01:14:24.288970 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 27 01:14:24.288970 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:14:24.288970 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 450/917] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:14:24 2022] GET /identity/v3/users/placement => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:14:24.297675 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e6fa80-be0b-4650-aa86-db19aa7da4c6 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:24.430935 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e6fa80-be0b-4650-aa86-db19aa7da4c6 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:24.472511 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58d970f-8cfb-4eb9-be42-c840de14c4b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:24.473251 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58d970f-8cfb-4eb9-be42-c840de14c4b5 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:24.473530 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58d970f-8cfb-4eb9-be42-c840de14c4b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:24.473821 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58d970f-8cfb-4eb9-be42-c840de14c4b5 None admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:24.474539 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58d970f-8cfb-4eb9-be42-c840de14c4b5 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:14:24.474845 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58d970f-8cfb-4eb9-be42-c840de14c4b5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:24.476463 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58d970f-8cfb-4eb9-be42-c840de14c4b5 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:24.499310 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 468/918] 10.222.0.42 () {56 vars in 1143 bytes} [Thu Oct 27 01:14:24 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:14:24.505983 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2069a339-c390-4632-939c-118e39deff32 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:24.640022 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2069a339-c390-4632-939c-118e39deff32 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:24.682710 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c56ecc-7d54-4306-a825-c1ba390c9fb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:24.683287 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c56ecc-7d54-4306-a825-c1ba390c9fb3 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:24.683582 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c56ecc-7d54-4306-a825-c1ba390c9fb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:24.683958 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c56ecc-7d54-4306-a825-c1ba390c9fb3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:24.696556 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c56ecc-7d54-4306-a825-c1ba390c9fb3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:24.698049 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c56ecc-7d54-4306-a825-c1ba390c9fb3 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:24.712921 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-06c56ecc-7d54-4306-a825-c1ba390c9fb3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:14:24.712921 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:14:24.712921 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:14:24.712921 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:14:24.712921 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:14:24.712921 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:14:24.712921 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:14:24.712921 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:14:24.712921 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:14:24.712921 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:14:24.712921 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:14:24.712921 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:14:24.712921 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:14:24.712921 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:14:24.712921 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:14:24.712921 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:14:24.714250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:14:24.714250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:14:24.714250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:14:24.714250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:14:24.714250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:14:24.714250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:14:24.714250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:14:24.714250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:14:24.714250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:14:24.714250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:14:24.714250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:14:24.714250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:14:24.714250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:14:24.714250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:14:24.714250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:14:24.714250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:14:24.714250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:14:24.715454 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:14:24.715454 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:14:24.715454 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:14:24.717216 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 451/919] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:14:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:14:24.721170 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58d970f-8cfb-4eb9-be42-c840de14c4b5 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:24.740962 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:24.850645 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58d970f-8cfb-4eb9-be42-c840de14c4b5 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:24.887323 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-346fc485-187b-4d4b-af58-a12d0af627a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:24.888779 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346fc485-187b-4d4b-af58-a12d0af627a3 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:24.889247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346fc485-187b-4d4b-af58-a12d0af627a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:24.889669 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346fc485-187b-4d4b-af58-a12d0af627a3 None admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:24.890514 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-346fc485-187b-4d4b-af58-a12d0af627a3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:14:24.890963 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-346fc485-187b-4d4b-af58-a12d0af627a3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:24.892464 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-346fc485-187b-4d4b-af58-a12d0af627a3 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:24.907130 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 469/920] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:14:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:14:24.913285 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c56ecc-7d54-4306-a825-c1ba390c9fb3 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:24.935706 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:25.030198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c56ecc-7d54-4306-a825-c1ba390c9fb3 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:25.067849 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc07878-1e6d-465a-b9b7-dbf576be62f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:25.068466 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc07878-1e6d-465a-b9b7-dbf576be62f2 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:25.068737 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc07878-1e6d-465a-b9b7-dbf576be62f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:25.069001 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc07878-1e6d-465a-b9b7-dbf576be62f2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:25.077377 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dc07878-1e6d-465a-b9b7-dbf576be62f2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:25.078987 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dc07878-1e6d-465a-b9b7-dbf576be62f2 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:25.090985 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-4dc07878-1e6d-465a-b9b7-dbf576be62f2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:14:25.090985 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:14:25.090985 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:14:25.090985 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:14:25.090985 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:14:25.090985 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:14:25.090985 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:14:25.090985 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:14:25.090985 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:14:25.090985 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:14:25.090985 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:14:25.090985 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:14:25.090985 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 27 01:14:25.090985 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 27 01:14:25.090985 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 27 01:14:25.092427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 27 01:14:25.092427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:14:25.092427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:14:25.092427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:14:25.092427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:14:25.092427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:14:25.092427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:14:25.092427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:14:25.092427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:14:25.092427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:14:25.092427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:14:25.092427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:14:25.092427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:14:25.092427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:14:25.092427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:14:25.092427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:14:25.092427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:14:25.093369 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 27 01:14:25.093369 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 27 01:14:25.093369 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 27 01:14:25.093369 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 27 01:14:25.093369 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 27 01:14:25.093369 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 27 01:14:25.093369 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:14:25.093369 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:14:25.093369 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 452/921] 10.222.0.42 () {56 vars in 1111 bytes} [Thu Oct 27 01:14:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 27 01:14:25.098378 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-346fc485-187b-4d4b-af58-a12d0af627a3 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:25.193087 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:25.221205 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-346fc485-187b-4d4b-af58-a12d0af627a3 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:25.263688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ccc725-44e1-459b-8752-c3cf6604f649 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:25.265776 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ccc725-44e1-459b-8752-c3cf6604f649 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:25.266084 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ccc725-44e1-459b-8752-c3cf6604f649 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:25.266400 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ccc725-44e1-459b-8752-c3cf6604f649 None admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:25.267251 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ccc725-44e1-459b-8752-c3cf6604f649 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:14:25.267698 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ccc725-44e1-459b-8752-c3cf6604f649 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:25.269380 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ccc725-44e1-459b-8752-c3cf6604f649 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:25.285726 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 470/922] 10.222.0.42 () {56 vars in 1148 bytes} [Thu Oct 27 01:14:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:14:25.291726 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc07878-1e6d-465a-b9b7-dbf576be62f2 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:25.376837 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:25.406988 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc07878-1e6d-465a-b9b7-dbf576be62f2 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:25.446216 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a94ba1-e6b5-4c8e-b1a0-a91cfe99a67d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:25.446843 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a94ba1-e6b5-4c8e-b1a0-a91cfe99a67d None admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:25.448276 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a94ba1-e6b5-4c8e-b1a0-a91cfe99a67d None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:25.448276 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a94ba1-e6b5-4c8e-b1a0-a91cfe99a67d None admin] PATH_INFO: `/v3/projects/23aad848d47c4d27ad4c138ff869e40f/users/62051562fc0b4cd08750d2293e71787c/roles/1b8f8dd9e8584053b048032db63254a4` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:25.465470 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a94ba1-e6b5-4c8e-b1a0-a91cfe99a67d None admin] RBAC: Authorizing `identity:create_grant(project_id=23aad848d47c4d27ad4c138ff869e40f, user_id=62051562fc0b4cd08750d2293e71787c, role_id=1b8f8dd9e8584053b048032db63254a4)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:25.467249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a94ba1-e6b5-4c8e-b1a0-a91cfe99a67d None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:25.485535 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 453/923] 10.222.0.42 () {58 vars in 1439 bytes} [Thu Oct 27 01:14:25 2022] PUT /identity/v3/projects/23aad848d47c4d27ad4c138ff869e40f/users/62051562fc0b4cd08750d2293e71787c/roles/1b8f8dd9e8584053b048032db63254a4 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:14:27.929046 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78789ffc-0f0c-4703-9e30-5fe4dcd12c5a None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:27.930024 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78789ffc-0f0c-4703-9e30-5fe4dcd12c5a None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:27.930024 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78789ffc-0f0c-4703-9e30-5fe4dcd12c5a None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:27.931785 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 471/924] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:14:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:14:27.938793 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b578cb-89d2-401c-8bea-4c51d8345750 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:27.939344 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b578cb-89d2-401c-8bea-4c51d8345750 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:27.939344 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b578cb-89d2-401c-8bea-4c51d8345750 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:27.959840 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-a1b578cb-89d2-401c-8bea-4c51d8345750 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:14:27.961037 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-a1b578cb-89d2-401c-8bea-4c51d8345750 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:14:28.027443 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 454/925] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:14:27 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 27 01:14:28.307966 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78789ffc-0f0c-4703-9e30-5fe4dcd12c5a None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:28.433339 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78789ffc-0f0c-4703-9e30-5fe4dcd12c5a None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:28.472095 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60412d33-ad43-45d1-b4b0-5449394f23ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:28.472766 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60412d33-ad43-45d1-b4b0-5449394f23ff None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:28.473173 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60412d33-ad43-45d1-b4b0-5449394f23ff None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:28.473500 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60412d33-ad43-45d1-b4b0-5449394f23ff None admin] PATH_INFO: `/v3/roles/service` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:28.481024 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:28.486714 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60412d33-ad43-45d1-b4b0-5449394f23ff None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:14:28.492423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-60412d33-ad43-45d1-b4b0-5449394f23ff None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 27 01:14:28.492423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:14:28.492423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:14:28.492423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:14:28.492423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:14:28.492423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:14:28.492423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:14:28.492423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:14:28.492423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:14:28.492423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:14:28.492423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:14:28.492423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:14:28.492423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:14:28.492423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:14:28.492423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:14:28.492423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:14:28.497214 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:14:28.497214 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:14:28.497214 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:14:28.497214 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:14:28.497214 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:14:28.497214 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:14:28.497214 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:14:28.497214 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:14:28.497214 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:14:28.497214 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:14:28.497214 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:14:28.497214 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:14:28.497214 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:14:28.497214 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:14:28.497214 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:14:28.497214 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:14:28.498554 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:14:28.498554 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:14:28.498554 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:14:28.498554 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:14:28.498554 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:14:28.498554 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:14:28.498554 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:14:28.498554 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:14:28.498554 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:14:28.498554 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:14:28.498554 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 27 01:14:28.498554 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:14:28.498554 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 472/926] 10.222.0.42 () {56 vars in 1102 bytes} [Thu Oct 27 01:14:28 2022] GET /identity/v3/roles/service => generated 85 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 27 01:14:28.499868 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b578cb-89d2-401c-8bea-4c51d8345750 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:28.562004 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:28.614359 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b578cb-89d2-401c-8bea-4c51d8345750 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:28.651710 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a723169-749c-4362-93f3-a36cee331d76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:28.652604 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a723169-749c-4362-93f3-a36cee331d76 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:28.652969 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a723169-749c-4362-93f3-a36cee331d76 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:28.654463 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a723169-749c-4362-93f3-a36cee331d76 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:28.655265 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a723169-749c-4362-93f3-a36cee331d76 None admin] RBAC: Adding query filter params (name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:14:28.655591 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a723169-749c-4362-93f3-a36cee331d76 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:28.656959 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a723169-749c-4362-93f3-a36cee331d76 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:28.666297 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 455/927] 10.222.0.42 () {56 vars in 1103 bytes} [Thu Oct 27 01:14:28 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:14:28.673395 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60412d33-ad43-45d1-b4b0-5449394f23ff None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:28.784388 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60412d33-ad43-45d1-b4b0-5449394f23ff None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:28.826287 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0322a1ef-5262-41cf-a149-3ac33e426d89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:28.826878 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0322a1ef-5262-41cf-a149-3ac33e426d89 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:28.827273 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0322a1ef-5262-41cf-a149-3ac33e426d89 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:28.827619 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0322a1ef-5262-41cf-a149-3ac33e426d89 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:28.838614 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0322a1ef-5262-41cf-a149-3ac33e426d89 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:28.840204 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0322a1ef-5262-41cf-a149-3ac33e426d89 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:28.858179 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-0322a1ef-5262-41cf-a149-3ac33e426d89 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:14:28.858179 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:14:28.858179 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:14:28.858179 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:14:28.858179 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:14:28.858179 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:14:28.858179 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:14:28.858179 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:14:28.858179 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:14:28.858179 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:14:28.858179 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:14:28.858179 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:14:28.858179 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:14:28.858179 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:14:28.858179 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:14:28.858179 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:14:28.859439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:14:28.859439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:14:28.859439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:14:28.859439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:14:28.859439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:14:28.859439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:14:28.859439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:14:28.859439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:14:28.859439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:14:28.859439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:14:28.859439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:14:28.859439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:14:28.859439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:14:28.859439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:14:28.859439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:14:28.859439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:14:28.859439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:14:28.860372 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:14:28.860372 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:14:28.860372 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:14:28.860372 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 473/928] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:14:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:14:28.865806 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a723169-749c-4362-93f3-a36cee331d76 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:28.978578 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a723169-749c-4362-93f3-a36cee331d76 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:28.998869 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:29.017946 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea8346b2-87e5-4bcc-b20d-7320e6d39a17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:29.018682 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8346b2-87e5-4bcc-b20d-7320e6d39a17 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:29.019021 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8346b2-87e5-4bcc-b20d-7320e6d39a17 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:29.019327 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8346b2-87e5-4bcc-b20d-7320e6d39a17 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:29.020070 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea8346b2-87e5-4bcc-b20d-7320e6d39a17 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:14:29.020445 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea8346b2-87e5-4bcc-b20d-7320e6d39a17 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:29.022090 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea8346b2-87e5-4bcc-b20d-7320e6d39a17 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:29.038008 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 456/929] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:14:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:14:29.043019 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0322a1ef-5262-41cf-a149-3ac33e426d89 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:29.099984 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:29.156285 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0322a1ef-5262-41cf-a149-3ac33e426d89 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:29.200812 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b5901f-1071-4aff-aed3-76a093f2e777 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:29.201609 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b5901f-1071-4aff-aed3-76a093f2e777 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:29.201983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b5901f-1071-4aff-aed3-76a093f2e777 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:29.202358 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b5901f-1071-4aff-aed3-76a093f2e777 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:29.212121 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b5901f-1071-4aff-aed3-76a093f2e777 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:29.213925 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b5901f-1071-4aff-aed3-76a093f2e777 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:29.226863 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-e8b5901f-1071-4aff-aed3-76a093f2e777 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 27 01:14:29.226863 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:14:29.226863 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:14:29.226863 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:14:29.226863 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:14:29.226863 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:14:29.226863 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:14:29.226863 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:14:29.226863 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:14:29.226863 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:14:29.226863 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:14:29.226863 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:14:29.226863 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 27 01:14:29.226863 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 27 01:14:29.226863 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 27 01:14:29.226863 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 27 01:14:29.228018 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:14:29.228018 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:14:29.228018 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 27 01:14:29.228018 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:14:29.228018 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 27 01:14:29.228018 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:14:29.228018 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:14:29.228018 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:14:29.228018 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:14:29.228018 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:14:29.228018 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:14:29.228018 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:14:29.228018 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:14:29.228018 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:14:29.228018 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:14:29.228018 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:14:29.228018 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:14:29.228018 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:14:29.229314 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:14:29.229314 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:14:29.229314 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 27 01:14:29.229314 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 27 01:14:29.229314 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 27 01:14:29.229314 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 27 01:14:29.229314 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 27 01:14:29.229314 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 27 01:14:29.229314 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 27 01:14:29.229314 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:14:29.229314 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 474/930] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:14:29 2022] GET /identity/v3/users/placement => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:14:29.233839 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea8346b2-87e5-4bcc-b20d-7320e6d39a17 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:29.345018 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea8346b2-87e5-4bcc-b20d-7320e6d39a17 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:29.396572 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f8a528a-d159-4aa2-94b0-db5eeed20cdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:29.397165 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8a528a-d159-4aa2-94b0-db5eeed20cdd None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:29.397511 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8a528a-d159-4aa2-94b0-db5eeed20cdd None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:29.397758 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8a528a-d159-4aa2-94b0-db5eeed20cdd None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:29.398579 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f8a528a-d159-4aa2-94b0-db5eeed20cdd None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:14:29.398921 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f8a528a-d159-4aa2-94b0-db5eeed20cdd None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:29.400580 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f8a528a-d159-4aa2-94b0-db5eeed20cdd None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:29.421353 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 457/931] 10.222.0.42 () {56 vars in 1143 bytes} [Thu Oct 27 01:14:29 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:14:29.427633 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b5901f-1071-4aff-aed3-76a093f2e777 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:29.537956 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b5901f-1071-4aff-aed3-76a093f2e777 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:29.550908 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:29.578552 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1c7514-3bff-49b7-a821-b2fcbbe6af13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:29.579138 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1c7514-3bff-49b7-a821-b2fcbbe6af13 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:29.579413 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1c7514-3bff-49b7-a821-b2fcbbe6af13 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:29.579688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1c7514-3bff-49b7-a821-b2fcbbe6af13 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:29.660444 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d1c7514-3bff-49b7-a821-b2fcbbe6af13 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:29.661989 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d1c7514-3bff-49b7-a821-b2fcbbe6af13 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:29.769970 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 475/932] 10.222.0.42 () {58 vars in 1282 bytes} [Thu Oct 27 01:14:29 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 27 01:14:29.776373 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f8a528a-d159-4aa2-94b0-db5eeed20cdd None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:29.814890 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:29.881785 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f8a528a-d159-4aa2-94b0-db5eeed20cdd None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:29.922120 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f105671-68ee-4278-b402-c3cecf17cfed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:29.922788 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f105671-68ee-4278-b402-c3cecf17cfed None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:29.923083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f105671-68ee-4278-b402-c3cecf17cfed None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:29.923385 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f105671-68ee-4278-b402-c3cecf17cfed None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:29.934300 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f105671-68ee-4278-b402-c3cecf17cfed None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:29.936080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f105671-68ee-4278-b402-c3cecf17cfed None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:29.954553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-6f105671-68ee-4278-b402-c3cecf17cfed None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:14:29.954553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:14:29.954553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:14:29.954553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:14:29.954553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:14:29.954553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:14:29.954553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:14:29.954553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:14:29.954553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:14:29.954553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:14:29.954553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:14:29.954553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:14:29.954553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:14:29.954553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:14:29.954553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:14:29.954553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:14:29.955781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:14:29.955781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:14:29.955781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:14:29.955781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:14:29.955781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:14:29.955781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:14:29.955781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:14:29.955781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:14:29.955781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:14:29.955781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:14:29.955781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:14:29.955781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:14:29.955781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:14:29.955781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:14:29.955781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:14:29.955781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:14:29.955781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:14:29.957102 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:14:29.957102 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:14:29.957102 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:14:29.957102 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 458/933] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:14:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 27 01:14:29.961797 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1c7514-3bff-49b7-a821-b2fcbbe6af13 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:29.975174 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:30.078093 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1c7514-3bff-49b7-a821-b2fcbbe6af13 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:30.116681 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f9d8fb-ae37-4121-bf74-fd7ae9a34bf6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:30.117422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f9d8fb-ae37-4121-bf74-fd7ae9a34bf6 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:30.117788 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f9d8fb-ae37-4121-bf74-fd7ae9a34bf6 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:30.118153 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f9d8fb-ae37-4121-bf74-fd7ae9a34bf6 None admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:30.118918 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f9d8fb-ae37-4121-bf74-fd7ae9a34bf6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:14:30.119332 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f9d8fb-ae37-4121-bf74-fd7ae9a34bf6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:30.120884 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f9d8fb-ae37-4121-bf74-fd7ae9a34bf6 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:30.132673 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 476/934] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:14:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:14:30.139361 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f105671-68ee-4278-b402-c3cecf17cfed None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:30.234412 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:30.249466 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f105671-68ee-4278-b402-c3cecf17cfed None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:30.284929 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b6fdc3-bb4b-446e-8db0-4a867c094f88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:30.285614 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b6fdc3-bb4b-446e-8db0-4a867c094f88 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:30.285874 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b6fdc3-bb4b-446e-8db0-4a867c094f88 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:30.286176 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b6fdc3-bb4b-446e-8db0-4a867c094f88 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:30.292783 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b6fdc3-bb4b-446e-8db0-4a867c094f88 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:30.295002 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b6fdc3-bb4b-446e-8db0-4a867c094f88 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:30.307969 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-c7b6fdc3-bb4b-446e-8db0-4a867c094f88 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:14:30.307969 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:14:30.307969 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:14:30.307969 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:14:30.307969 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:14:30.307969 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:14:30.307969 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:14:30.307969 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:14:30.307969 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:14:30.307969 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:14:30.307969 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:14:30.307969 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:14:30.307969 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 27 01:14:30.307969 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 27 01:14:30.307969 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 27 01:14:30.309492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 27 01:14:30.309492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:14:30.309492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:14:30.309492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:14:30.309492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:14:30.309492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:14:30.309492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:14:30.309492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:14:30.309492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:14:30.309492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:14:30.309492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:14:30.309492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:14:30.309492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:14:30.309492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:14:30.309492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:14:30.309492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:14:30.309492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:14:30.310831 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 27 01:14:30.310831 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 27 01:14:30.310831 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 27 01:14:30.310831 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 27 01:14:30.310831 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 27 01:14:30.310831 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 27 01:14:30.310831 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:14:30.310831 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:14:30.310831 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 459/935] 10.222.0.42 () {56 vars in 1111 bytes} [Thu Oct 27 01:14:30 2022] GET /identity/v3/projects/service => generated 88 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:14:30.316289 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f9d8fb-ae37-4121-bf74-fd7ae9a34bf6 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:30.402187 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:30.437916 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f9d8fb-ae37-4121-bf74-fd7ae9a34bf6 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:30.476086 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e38ddb-5df6-46c7-aa59-31df3e8cf896 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:30.476803 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e38ddb-5df6-46c7-aa59-31df3e8cf896 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:30.477154 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e38ddb-5df6-46c7-aa59-31df3e8cf896 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:30.477456 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e38ddb-5df6-46c7-aa59-31df3e8cf896 None admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:30.478192 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e38ddb-5df6-46c7-aa59-31df3e8cf896 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:14:30.478570 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e38ddb-5df6-46c7-aa59-31df3e8cf896 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:30.480346 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e38ddb-5df6-46c7-aa59-31df3e8cf896 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:30.494622 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 477/936] 10.222.0.42 () {56 vars in 1148 bytes} [Thu Oct 27 01:14:30 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:14:30.501904 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b6fdc3-bb4b-446e-8db0-4a867c094f88 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:30.614756 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b6fdc3-bb4b-446e-8db0-4a867c094f88 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:30.653618 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f50726f-ad51-4d7c-8425-bdcb060cd699 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:30.654259 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f50726f-ad51-4d7c-8425-bdcb060cd699 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:30.654623 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f50726f-ad51-4d7c-8425-bdcb060cd699 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:30.654933 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f50726f-ad51-4d7c-8425-bdcb060cd699 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:30.655730 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f50726f-ad51-4d7c-8425-bdcb060cd699 None admin] RBAC: Adding query filter params (role.id=1b8f8dd9e8584053b048032db63254a4, scope.project.id=23aad848d47c4d27ad4c138ff869e40f, user.id=62051562fc0b4cd08750d2293e71787c) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:14:30.656072 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f50726f-ad51-4d7c-8425-bdcb060cd699 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:30.657922 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f50726f-ad51-4d7c-8425-bdcb060cd699 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:30.664403 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 460/937] 10.222.0.42 () {56 vars in 1374 bytes} [Thu Oct 27 01:14:30 2022] GET /identity/v3/role_assignments?user.id=62051562fc0b4cd08750d2293e71787c&scope.project.id=23aad848d47c4d27ad4c138ff869e40f&role.id=1b8f8dd9e8584053b048032db63254a4 => generated 608 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:14:33.017448 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8869a6-375f-4e32-ba28-b3026fd54c7e None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:33.017998 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8869a6-375f-4e32-ba28-b3026fd54c7e None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:33.017998 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8869a6-375f-4e32-ba28-b3026fd54c7e None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:33.019290 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 478/938] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:14:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:14:33.024713 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02bd90e-8ce5-4f59-a2e0-22a59b1f81fa None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:33.025257 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02bd90e-8ce5-4f59-a2e0-22a59b1f81fa None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:33.025373 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02bd90e-8ce5-4f59-a2e0-22a59b1f81fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:33.045022 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-d02bd90e-8ce5-4f59-a2e0-22a59b1f81fa None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:14:33.046301 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d02bd90e-8ce5-4f59-a2e0-22a59b1f81fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:14:33.049551 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:33.119750 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 461/939] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:14:33 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:14:33.406165 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8869a6-375f-4e32-ba28-b3026fd54c7e None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:33.523066 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8869a6-375f-4e32-ba28-b3026fd54c7e None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:33.555424 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:33.562701 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1572728f-90f3-4060-a847-7b56136cc84e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:33.563514 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1572728f-90f3-4060-a847-7b56136cc84e None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:33.563892 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1572728f-90f3-4060-a847-7b56136cc84e None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:33.564262 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1572728f-90f3-4060-a847-7b56136cc84e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:33.576774 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1572728f-90f3-4060-a847-7b56136cc84e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:14:33.581770 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-1572728f-90f3-4060-a847-7b56136cc84e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:14:33.581770 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:14:33.581770 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:14:33.581770 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:14:33.581770 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:14:33.581770 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:14:33.581770 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:14:33.581770 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:14:33.581770 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:14:33.581770 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:14:33.581770 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:14:33.581770 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:14:33.581770 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:14:33.581770 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:14:33.581770 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:14:33.581770 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:14:33.582894 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:14:33.582894 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:14:33.582894 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:14:33.582894 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:14:33.582894 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:14:33.582894 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:14:33.582894 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:14:33.582894 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:14:33.582894 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:14:33.582894 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:14:33.582894 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:14:33.582894 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:14:33.582894 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:14:33.582894 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:14:33.582894 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:14:33.582894 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:14:33.584073 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:14:33.584073 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:14:33.584073 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:14:33.584073 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:14:33.584073 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:14:33.584073 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:14:33.584073 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:14:33.584073 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:14:33.584073 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:14:33.584073 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:14:33.584073 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:14:33.584073 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:14:33.584073 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 479/940] 10.222.0.42 () {56 vars in 1096 bytes} [Thu Oct 27 01:14:33 2022] GET /identity/v3/roles/admin => generated 83 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:14:33.589233 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02bd90e-8ce5-4f59-a2e0-22a59b1f81fa None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:33.705284 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02bd90e-8ce5-4f59-a2e0-22a59b1f81fa None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:33.740964 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06694f62-95a8-4828-83f8-e75011e12795 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:33.741820 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06694f62-95a8-4828-83f8-e75011e12795 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:33.742143 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06694f62-95a8-4828-83f8-e75011e12795 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:33.742516 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06694f62-95a8-4828-83f8-e75011e12795 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:33.743328 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06694f62-95a8-4828-83f8-e75011e12795 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:14:33.743743 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06694f62-95a8-4828-83f8-e75011e12795 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:33.745239 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06694f62-95a8-4828-83f8-e75011e12795 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:33.754649 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 462/941] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:14:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:14:33.760858 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1572728f-90f3-4060-a847-7b56136cc84e None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:33.873679 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1572728f-90f3-4060-a847-7b56136cc84e None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:33.909600 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77dd0721-420e-434b-b1c8-abb658f4eb38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:33.910283 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77dd0721-420e-434b-b1c8-abb658f4eb38 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:33.910624 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77dd0721-420e-434b-b1c8-abb658f4eb38 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:33.910874 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77dd0721-420e-434b-b1c8-abb658f4eb38 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:33.921407 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77dd0721-420e-434b-b1c8-abb658f4eb38 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:33.923005 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77dd0721-420e-434b-b1c8-abb658f4eb38 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:33.938505 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-77dd0721-420e-434b-b1c8-abb658f4eb38 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:14:33.938505 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:14:33.938505 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:14:33.938505 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:14:33.938505 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:14:33.938505 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:14:33.938505 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:14:33.938505 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:14:33.938505 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:14:33.938505 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:14:33.938505 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:14:33.938505 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:14:33.938505 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:14:33.938505 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:14:33.938505 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:14:33.938505 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:14:33.939672 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:14:33.939672 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:14:33.939672 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:14:33.939672 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:14:33.939672 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:14:33.939672 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:14:33.939672 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:14:33.939672 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:14:33.939672 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:14:33.939672 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:14:33.939672 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:14:33.939672 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:14:33.939672 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:14:33.939672 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:14:33.939672 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:14:33.939672 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:14:33.939672 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:14:33.940746 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:14:33.940746 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:14:33.940746 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:14:33.940746 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 480/942] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:14:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:14:33.945371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06694f62-95a8-4828-83f8-e75011e12795 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:33.948215 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:34.052682 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06694f62-95a8-4828-83f8-e75011e12795 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:34.092372 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034018c6-86e0-4e45-bd48-1cf37c5cb34b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:34.093107 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034018c6-86e0-4e45-bd48-1cf37c5cb34b None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:34.093394 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034018c6-86e0-4e45-bd48-1cf37c5cb34b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:34.093756 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034018c6-86e0-4e45-bd48-1cf37c5cb34b None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:34.094313 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-034018c6-86e0-4e45-bd48-1cf37c5cb34b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:14:34.094613 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-034018c6-86e0-4e45-bd48-1cf37c5cb34b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:34.095971 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-034018c6-86e0-4e45-bd48-1cf37c5cb34b None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:34.108393 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 463/943] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:14:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:14:34.114537 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77dd0721-420e-434b-b1c8-abb658f4eb38 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:34.163139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:34.227366 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77dd0721-420e-434b-b1c8-abb658f4eb38 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:34.264877 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b7e447-c2ed-4ee4-b874-767b402c2497 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:34.265548 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b7e447-c2ed-4ee4-b874-767b402c2497 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:34.265868 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b7e447-c2ed-4ee4-b874-767b402c2497 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:34.266187 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b7e447-c2ed-4ee4-b874-767b402c2497 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:34.274598 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b7e447-c2ed-4ee4-b874-767b402c2497 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:34.276145 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b7e447-c2ed-4ee4-b874-767b402c2497 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:34.288343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-01b7e447-c2ed-4ee4-b874-767b402c2497 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 27 01:14:34.288343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:14:34.288343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:14:34.288343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:14:34.288343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:14:34.288343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:14:34.288343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:14:34.288343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:14:34.288343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:14:34.288343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:14:34.288343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:14:34.288343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:14:34.288343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 27 01:14:34.288343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 27 01:14:34.288343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 27 01:14:34.288343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 27 01:14:34.289396 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:14:34.289396 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:14:34.289396 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 27 01:14:34.289396 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:14:34.289396 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 27 01:14:34.289396 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:14:34.289396 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:14:34.289396 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:14:34.289396 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:14:34.289396 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:14:34.289396 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:14:34.289396 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:14:34.289396 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:14:34.289396 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:14:34.289396 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:14:34.289396 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:14:34.289396 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:14:34.289396 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:14:34.290369 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:14:34.290369 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:14:34.290369 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 27 01:14:34.290369 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 27 01:14:34.290369 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 27 01:14:34.290369 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 27 01:14:34.290369 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 27 01:14:34.290369 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 27 01:14:34.290369 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 27 01:14:34.290369 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:14:34.290369 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 481/944] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:14:34 2022] GET /identity/v3/users/placement => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 27 01:14:34.295693 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034018c6-86e0-4e45-bd48-1cf37c5cb34b None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:34.364553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:34.412151 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034018c6-86e0-4e45-bd48-1cf37c5cb34b None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:34.455758 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d636fb23-a0ed-4206-a17b-ba8359389e7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:34.456451 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d636fb23-a0ed-4206-a17b-ba8359389e7c None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:34.456759 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d636fb23-a0ed-4206-a17b-ba8359389e7c None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:34.457037 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d636fb23-a0ed-4206-a17b-ba8359389e7c None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:34.457918 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d636fb23-a0ed-4206-a17b-ba8359389e7c None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:14:34.458278 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d636fb23-a0ed-4206-a17b-ba8359389e7c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:34.459880 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d636fb23-a0ed-4206-a17b-ba8359389e7c None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:34.479693 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 464/945] 10.222.0.42 () {56 vars in 1143 bytes} [Thu Oct 27 01:14:34 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:14:34.486515 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b7e447-c2ed-4ee4-b874-767b402c2497 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:34.595423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b7e447-c2ed-4ee4-b874-767b402c2497 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:34.597208 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:34.633330 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78629cf6-54e7-405f-9910-ee812e804b95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:34.634043 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78629cf6-54e7-405f-9910-ee812e804b95 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:34.634431 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78629cf6-54e7-405f-9910-ee812e804b95 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:34.634794 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78629cf6-54e7-405f-9910-ee812e804b95 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:34.645555 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78629cf6-54e7-405f-9910-ee812e804b95 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:34.647115 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78629cf6-54e7-405f-9910-ee812e804b95 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:34.663310 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-78629cf6-54e7-405f-9910-ee812e804b95 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:14:34.663310 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:14:34.663310 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:14:34.663310 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:14:34.663310 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:14:34.663310 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:14:34.663310 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:14:34.663310 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:14:34.663310 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:14:34.663310 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:14:34.663310 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:14:34.663310 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:14:34.663310 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:14:34.663310 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:14:34.663310 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:14:34.663310 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:14:34.664507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:14:34.664507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:14:34.664507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:14:34.664507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:14:34.664507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:14:34.664507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:14:34.664507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:14:34.664507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:14:34.664507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:14:34.664507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:14:34.664507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:14:34.664507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:14:34.664507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:14:34.664507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:14:34.664507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:14:34.664507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:14:34.664507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:14:34.665653 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:14:34.665653 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:14:34.665653 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:14:34.665653 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 482/946] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:14:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:14:34.671376 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d636fb23-a0ed-4206-a17b-ba8359389e7c None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:34.773358 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d636fb23-a0ed-4206-a17b-ba8359389e7c None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:34.795930 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:34.814122 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2c1a2d-acad-432e-9701-dddd99ae3900 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:34.814827 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2c1a2d-acad-432e-9701-dddd99ae3900 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:34.815107 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2c1a2d-acad-432e-9701-dddd99ae3900 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:34.815384 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2c1a2d-acad-432e-9701-dddd99ae3900 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:34.816026 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f2c1a2d-acad-432e-9701-dddd99ae3900 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:14:34.816332 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f2c1a2d-acad-432e-9701-dddd99ae3900 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:34.817736 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f2c1a2d-acad-432e-9701-dddd99ae3900 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:34.830140 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 465/947] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:14:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:14:34.836363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78629cf6-54e7-405f-9910-ee812e804b95 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:34.955639 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78629cf6-54e7-405f-9910-ee812e804b95 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:34.993360 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc5d9f0-eabb-4256-b5a0-6a2d900823b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:34.994140 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc5d9f0-eabb-4256-b5a0-6a2d900823b7 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:34.994524 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc5d9f0-eabb-4256-b5a0-6a2d900823b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:34.994889 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc5d9f0-eabb-4256-b5a0-6a2d900823b7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:35.002619 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc5d9f0-eabb-4256-b5a0-6a2d900823b7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:35.004298 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc5d9f0-eabb-4256-b5a0-6a2d900823b7 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:35.017755 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-fbc5d9f0-eabb-4256-b5a0-6a2d900823b7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:14:35.017755 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:14:35.017755 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:14:35.017755 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:14:35.017755 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:14:35.017755 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:14:35.017755 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:14:35.017755 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:14:35.017755 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:14:35.017755 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:14:35.017755 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:14:35.017755 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:14:35.017755 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 27 01:14:35.017755 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 27 01:14:35.017755 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 27 01:14:35.018908 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 27 01:14:35.018908 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:14:35.018908 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:14:35.018908 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:14:35.018908 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:14:35.018908 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:14:35.018908 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:14:35.018908 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:14:35.018908 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:14:35.018908 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:14:35.018908 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:14:35.018908 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:14:35.018908 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:14:35.018908 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:14:35.018908 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:14:35.018908 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:14:35.018908 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:14:35.019966 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 27 01:14:35.019966 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 27 01:14:35.019966 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 27 01:14:35.019966 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 27 01:14:35.019966 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 27 01:14:35.019966 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 27 01:14:35.019966 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:14:35.019966 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:14:35.019966 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 483/948] 10.222.0.42 () {56 vars in 1111 bytes} [Thu Oct 27 01:14:34 2022] GET /identity/v3/projects/service => generated 88 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 27 01:14:35.025713 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2c1a2d-acad-432e-9701-dddd99ae3900 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:35.135766 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2c1a2d-acad-432e-9701-dddd99ae3900 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:35.174182 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a3a2b1-1d91-4ca5-a11f-fcf31ce7ff49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:35.174836 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a3a2b1-1d91-4ca5-a11f-fcf31ce7ff49 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:35.175113 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a3a2b1-1d91-4ca5-a11f-fcf31ce7ff49 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:35.175451 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a3a2b1-1d91-4ca5-a11f-fcf31ce7ff49 None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:35.176185 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29a3a2b1-1d91-4ca5-a11f-fcf31ce7ff49 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:14:35.176506 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29a3a2b1-1d91-4ca5-a11f-fcf31ce7ff49 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:35.178018 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29a3a2b1-1d91-4ca5-a11f-fcf31ce7ff49 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:35.190998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 466/949] 10.222.0.42 () {56 vars in 1148 bytes} [Thu Oct 27 01:14:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:14:35.197725 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc5d9f0-eabb-4256-b5a0-6a2d900823b7 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:35.216476 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:35.310115 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc5d9f0-eabb-4256-b5a0-6a2d900823b7 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:35.348710 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f038287a-62e2-4a64-b5ff-982dee41309a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:35.349436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f038287a-62e2-4a64-b5ff-982dee41309a None admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:35.349710 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f038287a-62e2-4a64-b5ff-982dee41309a None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:35.349988 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f038287a-62e2-4a64-b5ff-982dee41309a None admin] PATH_INFO: `/v3/projects/23aad848d47c4d27ad4c138ff869e40f/users/62051562fc0b4cd08750d2293e71787c/roles/414286927fdd44688107ff5e10235acf` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:35.351862 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f038287a-62e2-4a64-b5ff-982dee41309a None admin] RBAC: Authorizing `identity:create_grant(project_id=23aad848d47c4d27ad4c138ff869e40f, user_id=62051562fc0b4cd08750d2293e71787c, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:35.353318 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f038287a-62e2-4a64-b5ff-982dee41309a None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:35.370094 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 484/950] 10.222.0.42 () {58 vars in 1439 bytes} [Thu Oct 27 01:14:35 2022] PUT /identity/v3/projects/23aad848d47c4d27ad4c138ff869e40f/users/62051562fc0b4cd08750d2293e71787c/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:14:37.919689 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0099611d-1161-4ead-89a2-7506d404d37f None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:37.920255 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0099611d-1161-4ead-89a2-7506d404d37f None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:37.920398 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0099611d-1161-4ead-89a2-7506d404d37f None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:37.922105 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 467/951] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:14:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:14:37.927872 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204aad52-37f9-4341-a96b-9f6bcd10c809 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:37.928344 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204aad52-37f9-4341-a96b-9f6bcd10c809 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:37.928439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204aad52-37f9-4341-a96b-9f6bcd10c809 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:37.948295 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-204aad52-37f9-4341-a96b-9f6bcd10c809 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:14:37.950042 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-204aad52-37f9-4341-a96b-9f6bcd10c809 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:14:38.021208 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 485/952] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:14:37 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:14:38.291816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0099611d-1161-4ead-89a2-7506d404d37f None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:38.339438 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:38.405687 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0099611d-1161-4ead-89a2-7506d404d37f None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:38.446101 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf90a13-9e04-4b74-b523-43861a5eca9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:38.446731 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf90a13-9e04-4b74-b523-43861a5eca9d None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:38.447006 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf90a13-9e04-4b74-b523-43861a5eca9d None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:38.447274 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf90a13-9e04-4b74-b523-43861a5eca9d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:38.458614 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf90a13-9e04-4b74-b523-43861a5eca9d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 27 01:14:38.464262 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-5bf90a13-9e04-4b74-b523-43861a5eca9d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:14:38.464262 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:14:38.464262 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:14:38.464262 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:14:38.464262 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:14:38.464262 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:14:38.464262 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:14:38.464262 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:14:38.464262 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:14:38.464262 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:14:38.464262 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:14:38.464262 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:14:38.464262 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 27 01:14:38.464262 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 27 01:14:38.464262 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 27 01:14:38.464262 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 27 01:14:38.465318 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 27 01:14:38.465318 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 27 01:14:38.465318 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 27 01:14:38.465318 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 27 01:14:38.465318 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:14:38.465318 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:14:38.465318 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:14:38.465318 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:14:38.465318 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:14:38.465318 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:14:38.465318 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:14:38.465318 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:14:38.465318 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:14:38.465318 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:14:38.465318 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:14:38.465318 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:14:38.467169 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:14:38.467169 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:14:38.467169 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:14:38.467169 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:14:38.467169 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 27 01:14:38.467169 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 27 01:14:38.467169 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 27 01:14:38.467169 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 27 01:14:38.467169 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 27 01:14:38.467169 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 27 01:14:38.467169 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 27 01:14:38.467169 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:14:38.467169 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 468/953] 10.222.0.42 () {56 vars in 1096 bytes} [Thu Oct 27 01:14:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:14:38.472364 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-204aad52-37f9-4341-a96b-9f6bcd10c809 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:38.476017 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:38.580092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-204aad52-37f9-4341-a96b-9f6bcd10c809 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:38.619595 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c414d5ab-ac50-430c-8809-6741370ad657 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:38.620197 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c414d5ab-ac50-430c-8809-6741370ad657 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:38.620474 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c414d5ab-ac50-430c-8809-6741370ad657 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:38.620745 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c414d5ab-ac50-430c-8809-6741370ad657 None admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:38.621507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c414d5ab-ac50-430c-8809-6741370ad657 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:14:38.622819 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c414d5ab-ac50-430c-8809-6741370ad657 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:38.623970 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c414d5ab-ac50-430c-8809-6741370ad657 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:38.633884 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 486/954] 10.222.0.42 () {56 vars in 1099 bytes} [Thu Oct 27 01:14:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:14:38.640859 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf90a13-9e04-4b74-b523-43861a5eca9d None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:38.750794 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:38.753664 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf90a13-9e04-4b74-b523-43861a5eca9d None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:38.791588 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49290020-bb92-4a6f-b8c6-e3ff794934c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:38.792381 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49290020-bb92-4a6f-b8c6-e3ff794934c3 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:38.792696 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49290020-bb92-4a6f-b8c6-e3ff794934c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:38.793077 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49290020-bb92-4a6f-b8c6-e3ff794934c3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:38.806073 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49290020-bb92-4a6f-b8c6-e3ff794934c3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:38.807539 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49290020-bb92-4a6f-b8c6-e3ff794934c3 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:38.823521 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-49290020-bb92-4a6f-b8c6-e3ff794934c3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:14:38.823521 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:14:38.823521 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:14:38.823521 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:14:38.823521 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:14:38.823521 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:14:38.823521 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:14:38.823521 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:14:38.823521 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:14:38.823521 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:14:38.823521 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:14:38.823521 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:14:38.823521 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:14:38.823521 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:14:38.823521 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:14:38.823521 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:14:38.825188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:14:38.825188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:14:38.825188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:14:38.825188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:14:38.825188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:14:38.825188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:14:38.825188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:14:38.825188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:14:38.825188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:14:38.825188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:14:38.825188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:14:38.825188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:14:38.825188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:14:38.825188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:14:38.825188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:14:38.825188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:14:38.825188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:14:38.826303 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:14:38.826303 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:14:38.826303 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:14:38.826303 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 469/955] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:14:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 27 01:14:38.830970 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c414d5ab-ac50-430c-8809-6741370ad657 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:38.907781 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:38.946232 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c414d5ab-ac50-430c-8809-6741370ad657 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:38.984776 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c02274-c480-4fe9-a230-36e866cea9cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:38.985412 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c02274-c480-4fe9-a230-36e866cea9cb None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:38.985704 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c02274-c480-4fe9-a230-36e866cea9cb None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:38.985952 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c02274-c480-4fe9-a230-36e866cea9cb None admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:38.986578 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c02274-c480-4fe9-a230-36e866cea9cb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:14:38.986871 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c02274-c480-4fe9-a230-36e866cea9cb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:38.988247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c02274-c480-4fe9-a230-36e866cea9cb None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:39.001444 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 487/956] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:14:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:14:39.007303 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49290020-bb92-4a6f-b8c6-e3ff794934c3 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:39.110799 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49290020-bb92-4a6f-b8c6-e3ff794934c3 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:39.144910 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3accdf7-7258-4a8b-b898-c2dd4f4cfbdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:39.145573 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3accdf7-7258-4a8b-b898-c2dd4f4cfbdd None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:39.145854 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3accdf7-7258-4a8b-b898-c2dd4f4cfbdd None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:39.146207 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3accdf7-7258-4a8b-b898-c2dd4f4cfbdd None admin] PATH_INFO: `/v3/users/placement` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:39.155311 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3accdf7-7258-4a8b-b898-c2dd4f4cfbdd None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:39.156741 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3accdf7-7258-4a8b-b898-c2dd4f4cfbdd None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:39.168130 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-e3accdf7-7258-4a8b-b898-c2dd4f4cfbdd None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 27 01:14:39.168130 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:14:39.168130 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:14:39.168130 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:14:39.168130 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:14:39.168130 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:14:39.168130 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:14:39.168130 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:14:39.168130 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:14:39.168130 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:14:39.168130 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:14:39.168130 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:14:39.168130 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 27 01:14:39.168130 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 27 01:14:39.168130 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 27 01:14:39.168130 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 27 01:14:39.169399 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:14:39.169399 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:14:39.169399 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 27 01:14:39.169399 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:14:39.169399 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 27 01:14:39.169399 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 27 01:14:39.169399 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:14:39.169399 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:14:39.169399 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:14:39.169399 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:14:39.169399 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:14:39.169399 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:14:39.169399 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:14:39.169399 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:14:39.169399 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:14:39.169399 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:14:39.169399 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:14:39.169399 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:14:39.170797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:14:39.170797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:14:39.170797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 27 01:14:39.170797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 27 01:14:39.170797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 27 01:14:39.170797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 27 01:14:39.170797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 27 01:14:39.170797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 27 01:14:39.170797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 27 01:14:39.170797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:14:39.170797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 470/957] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:14:39 2022] GET /identity/v3/users/placement => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:14:39.175566 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c02274-c480-4fe9-a230-36e866cea9cb None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:39.286574 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c02274-c480-4fe9-a230-36e866cea9cb None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:39.316191 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:39.326680 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b1b2c9-5f65-40bd-b8a4-b38199aef618 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:39.327322 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b1b2c9-5f65-40bd-b8a4-b38199aef618 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:39.327603 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b1b2c9-5f65-40bd-b8a4-b38199aef618 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:39.327880 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b1b2c9-5f65-40bd-b8a4-b38199aef618 None admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:39.328646 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08b1b2c9-5f65-40bd-b8a4-b38199aef618 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:14:39.328953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08b1b2c9-5f65-40bd-b8a4-b38199aef618 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:39.330550 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08b1b2c9-5f65-40bd-b8a4-b38199aef618 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:39.354409 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 488/958] 10.222.0.42 () {56 vars in 1143 bytes} [Thu Oct 27 01:14:39 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:14:39.355652 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3accdf7-7258-4a8b-b898-c2dd4f4cfbdd None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:39.374438 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:39.488719 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3accdf7-7258-4a8b-b898-c2dd4f4cfbdd None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:39.525688 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa9689e3-b863-42e3-9acc-6aa01151b633 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:39.526321 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9689e3-b863-42e3-9acc-6aa01151b633 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:39.526596 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9689e3-b863-42e3-9acc-6aa01151b633 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:39.526864 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9689e3-b863-42e3-9acc-6aa01151b633 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:39.601785 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa9689e3-b863-42e3-9acc-6aa01151b633 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:39.603250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa9689e3-b863-42e3-9acc-6aa01151b633 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:39.617641 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:39.707537 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 471/959] 10.222.0.42 () {58 vars in 1282 bytes} [Thu Oct 27 01:14:39 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 27 01:14:39.713576 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b1b2c9-5f65-40bd-b8a4-b38199aef618 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:39.819886 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b1b2c9-5f65-40bd-b8a4-b38199aef618 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:39.856867 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e3cf05-93d5-4b81-8d8a-90418c1c5991 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:39.857529 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e3cf05-93d5-4b81-8d8a-90418c1c5991 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:39.857806 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e3cf05-93d5-4b81-8d8a-90418c1c5991 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:39.858075 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e3cf05-93d5-4b81-8d8a-90418c1c5991 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:39.868559 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e3cf05-93d5-4b81-8d8a-90418c1c5991 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:39.870042 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e3cf05-93d5-4b81-8d8a-90418c1c5991 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:39.885873 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-18e3cf05-93d5-4b81-8d8a-90418c1c5991 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:14:39.885873 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:14:39.885873 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:14:39.885873 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:14:39.885873 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:14:39.885873 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:14:39.885873 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:14:39.885873 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:14:39.885873 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:14:39.885873 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:14:39.885873 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:14:39.885873 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:14:39.885873 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 27 01:14:39.885873 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 27 01:14:39.885873 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 27 01:14:39.885873 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 27 01:14:39.887127 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:14:39.887127 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:14:39.887127 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:14:39.887127 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:14:39.887127 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:14:39.887127 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:14:39.887127 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:14:39.887127 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:14:39.887127 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:14:39.887127 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:14:39.887127 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:14:39.887127 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:14:39.887127 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:14:39.887127 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:14:39.887127 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:14:39.887127 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:14:39.887127 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 27 01:14:39.893899 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 27 01:14:39.893899 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 27 01:14:39.893899 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:14:39.893899 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 489/960] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:14:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:14:39.894636 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa9689e3-b863-42e3-9acc-6aa01151b633 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:39.995452 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa9689e3-b863-42e3-9acc-6aa01151b633 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:40.009449 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:40.031730 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6b7acf-61e9-4fcd-8f06-7c1f117dad2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:40.032332 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6b7acf-61e9-4fcd-8f06-7c1f117dad2d None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:40.032607 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6b7acf-61e9-4fcd-8f06-7c1f117dad2d None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:40.032880 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6b7acf-61e9-4fcd-8f06-7c1f117dad2d None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:40.033562 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc6b7acf-61e9-4fcd-8f06-7c1f117dad2d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:14:40.033861 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc6b7acf-61e9-4fcd-8f06-7c1f117dad2d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:40.035206 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc6b7acf-61e9-4fcd-8f06-7c1f117dad2d None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:40.046389 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 472/961] 10.222.0.42 () {56 vars in 1109 bytes} [Thu Oct 27 01:14:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:14:40.052132 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e3cf05-93d5-4b81-8d8a-90418c1c5991 None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:40.092146 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:40.159585 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e3cf05-93d5-4b81-8d8a-90418c1c5991 None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:40.194153 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802c035e-bdea-440f-95b4-6a95bc5c183d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:40.194772 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802c035e-bdea-440f-95b4-6a95bc5c183d None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:40.195044 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802c035e-bdea-440f-95b4-6a95bc5c183d None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:40.195337 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802c035e-bdea-440f-95b4-6a95bc5c183d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:40.202319 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-802c035e-bdea-440f-95b4-6a95bc5c183d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:40.203757 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-802c035e-bdea-440f-95b4-6a95bc5c183d None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:40.214376 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application [None req-802c035e-bdea-440f-95b4-6a95bc5c183d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:14:40.214376 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:14:40.214376 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:14:40.214376 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:14:40.214376 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:14:40.214376 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:14:40.214376 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:14:40.214376 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:14:40.214376 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:14:40.214376 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:14:40.214376 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:14:40.214376 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:14:40.214376 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 27 01:14:40.214376 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 27 01:14:40.214376 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 27 01:14:40.216119 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 27 01:14:40.216119 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:14:40.216119 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:14:40.216119 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:14:40.216119 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:14:40.216119 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:14:40.216119 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:14:40.216119 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:14:40.216119 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application with Lock( Oct 27 01:14:40.216119 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:14:40.216119 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:14:40.216119 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:14:40.216119 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:14:40.216119 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:14:40.216119 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:14:40.216119 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:14:40.216119 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:14:40.217252 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 27 01:14:40.217252 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 27 01:14:40.217252 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 27 01:14:40.217252 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 27 01:14:40.217252 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 27 01:14:40.217252 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 27 01:14:40.217252 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 27 01:14:40.217252 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: ERROR keystone.server.flask.application Oct 27 01:14:40.217252 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 490/962] 10.222.0.42 () {56 vars in 1111 bytes} [Thu Oct 27 01:14:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:14:40.221900 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6b7acf-61e9-4fcd-8f06-7c1f117dad2d None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:40.317887 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6b7acf-61e9-4fcd-8f06-7c1f117dad2d None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:40.350406 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e07082-537a-4a25-87db-64912d919850 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:40.350989 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e07082-537a-4a25-87db-64912d919850 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:40.351264 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e07082-537a-4a25-87db-64912d919850 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:40.351532 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e07082-537a-4a25-87db-64912d919850 None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:40.352205 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43e07082-537a-4a25-87db-64912d919850 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:14:40.352501 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43e07082-537a-4a25-87db-64912d919850 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:40.353936 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43e07082-537a-4a25-87db-64912d919850 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:40.365237 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 473/963] 10.222.0.42 () {56 vars in 1148 bytes} [Thu Oct 27 01:14:40 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:14:40.371914 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802c035e-bdea-440f-95b4-6a95bc5c183d None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:40.478075 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:40.484377 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802c035e-bdea-440f-95b4-6a95bc5c183d None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:40.517940 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa78a84-de83-4a92-bd4b-144761e7b4e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:40.518522 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa78a84-de83-4a92-bd4b-144761e7b4e4 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:40.518794 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa78a84-de83-4a92-bd4b-144761e7b4e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:40.519058 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa78a84-de83-4a92-bd4b-144761e7b4e4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:40.519764 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caa78a84-de83-4a92-bd4b-144761e7b4e4 None admin] RBAC: Adding query filter params (role.id=414286927fdd44688107ff5e10235acf, scope.project.id=23aad848d47c4d27ad4c138ff869e40f, user.id=62051562fc0b4cd08750d2293e71787c) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:14:40.520066 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caa78a84-de83-4a92-bd4b-144761e7b4e4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:40.521507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caa78a84-de83-4a92-bd4b-144761e7b4e4 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:40.526970 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 491/964] 10.222.0.42 () {56 vars in 1374 bytes} [Thu Oct 27 01:14:40 2022] GET /identity/v3/role_assignments?user.id=62051562fc0b4cd08750d2293e71787c&scope.project.id=23aad848d47c4d27ad4c138ff869e40f&role.id=414286927fdd44688107ff5e10235acf => generated 608 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:14:42.687391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c848b02b-b939-4194-928d-7b00cf5468ce None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:42.687924 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c848b02b-b939-4194-928d-7b00cf5468ce None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:42.687924 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c848b02b-b939-4194-928d-7b00cf5468ce None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:42.689291 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 474/965] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:14:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:14:42.694491 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce9607b-be34-48c2-aff9-f17ea0576777 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:42.694866 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce9607b-be34-48c2-aff9-f17ea0576777 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:42.695035 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce9607b-be34-48c2-aff9-f17ea0576777 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:42.713339 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-4ce9607b-be34-48c2-aff9-f17ea0576777 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:14:42.714518 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-4ce9607b-be34-48c2-aff9-f17ea0576777 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:14:42.778842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 492/966] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:14:42 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:14:43.047734 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c848b02b-b939-4194-928d-7b00cf5468ce None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:43.078839 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:43.146552 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c848b02b-b939-4194-928d-7b00cf5468ce None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:43.179964 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42eaec9-711b-4d1f-9dcd-1816c778fa25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:43.180571 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42eaec9-711b-4d1f-9dcd-1816c778fa25 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:43.180848 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42eaec9-711b-4d1f-9dcd-1816c778fa25 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:43.181146 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42eaec9-711b-4d1f-9dcd-1816c778fa25 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:43.181718 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a42eaec9-711b-4d1f-9dcd-1816c778fa25 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:43.183063 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a42eaec9-711b-4d1f-9dcd-1816c778fa25 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:43.192148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-a42eaec9-711b-4d1f-9dcd-1816c778fa25 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 27 01:14:43.192148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:14:43.192148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 27 01:14:43.192148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 27 01:14:43.192148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 27 01:14:43.192148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 27 01:14:43.192148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 27 01:14:43.192148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:14:43.192148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 27 01:14:43.192148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:14:43.192148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 27 01:14:43.192148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:14:43.192148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:14:43.192148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:14:43.192148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:14:43.192148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:14:43.192148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:14:43.193357 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:14:43.193357 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:14:43.193357 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:14:43.193357 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:14:43.193357 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:14:43.193357 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:14:43.193357 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 27 01:14:43.193357 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 27 01:14:43.193357 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 27 01:14:43.193357 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 27 01:14:43.193357 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:14:43.193357 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:14:43.193357 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:14:43.193357 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:14:43.193357 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:14:43.193357 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:14:43.194401 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:14:43.194401 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:14:43.194401 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:14:43.194401 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:14:43.194401 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:14:43.194401 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:14:43.194401 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:14:43.194401 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:14:43.194401 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:14:43.194401 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:14:43.194401 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 27 01:14:43.194401 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:14:43.194401 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 27 01:14:43.194401 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:14:43.194401 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 475/967] 10.222.0.42 () {56 vars in 1117 bytes} [Thu Oct 27 01:14:43 2022] GET /identity/v3/services/placement => generated 90 bytes in 146 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:14:43.199170 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce9607b-be34-48c2-aff9-f17ea0576777 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:43.284103 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:43.294304 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce9607b-be34-48c2-aff9-f17ea0576777 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:43.330245 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a91843-4b6f-4345-b3ca-4406a00815b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:43.330783 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a91843-4b6f-4345-b3ca-4406a00815b7 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:43.331060 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a91843-4b6f-4345-b3ca-4406a00815b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:43.331330 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a91843-4b6f-4345-b3ca-4406a00815b7 None admin] PATH_INFO: `/v3/services` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:43.331947 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a91843-4b6f-4345-b3ca-4406a00815b7 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:14:43.332246 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a91843-4b6f-4345-b3ca-4406a00815b7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:43.333641 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a91843-4b6f-4345-b3ca-4406a00815b7 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:43.338261 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 493/968] 10.222.0.42 () {56 vars in 1116 bytes} [Thu Oct 27 01:14:43 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:14:43.344224 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42eaec9-711b-4d1f-9dcd-1816c778fa25 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:43.451526 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:43.457762 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42eaec9-711b-4d1f-9dcd-1816c778fa25 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:43.490388 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f7bbbca-ef0b-4c2f-84ec-ff736cce69c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:43.490969 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7bbbca-ef0b-4c2f-84ec-ff736cce69c4 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:43.491240 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7bbbca-ef0b-4c2f-84ec-ff736cce69c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:43.491502 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7bbbca-ef0b-4c2f-84ec-ff736cce69c4 None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:43.492117 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f7bbbca-ef0b-4c2f-84ec-ff736cce69c4 None admin] RBAC: Adding query filter params (type=placement) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:14:43.492408 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f7bbbca-ef0b-4c2f-84ec-ff736cce69c4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:43.493777 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f7bbbca-ef0b-4c2f-84ec-ff736cce69c4 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:43.498315 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 476/969] 10.222.0.42 () {56 vars in 1116 bytes} [Thu Oct 27 01:14:43 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:14:45.647347 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9648881c-da3c-4279-9530-141b2e50968e None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:45.647911 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9648881c-da3c-4279-9530-141b2e50968e None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:45.647911 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9648881c-da3c-4279-9530-141b2e50968e None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:45.649229 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 494/970] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:14:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:14:45.655587 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51dca95-5bb9-40cc-a259-a91d8c8847f9 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:45.656019 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51dca95-5bb9-40cc-a259-a91d8c8847f9 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:45.656196 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51dca95-5bb9-40cc-a259-a91d8c8847f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:45.674127 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-d51dca95-5bb9-40cc-a259-a91d8c8847f9 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:14:45.675311 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d51dca95-5bb9-40cc-a259-a91d8c8847f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:14:45.739337 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 477/971] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:14:45 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:14:46.007669 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9648881c-da3c-4279-9530-141b2e50968e None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:46.106753 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9648881c-da3c-4279-9530-141b2e50968e None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:46.139435 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090486cf-ff74-429e-866b-6712f183bf32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:46.140025 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090486cf-ff74-429e-866b-6712f183bf32 None admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:46.140302 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090486cf-ff74-429e-866b-6712f183bf32 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:46.140571 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090486cf-ff74-429e-866b-6712f183bf32 None admin] PATH_INFO: `/v3/services` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:46.141247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090486cf-ff74-429e-866b-6712f183bf32 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:46.142591 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090486cf-ff74-429e-866b-6712f183bf32 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:46.156272 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 495/972] 10.222.0.42 () {60 vars in 1141 bytes} [Thu Oct 27 01:14:46 2022] POST /identity/v3/services => generated 246 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:14:48.360565 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b211d1-df7f-4127-b91d-4395564b4e56 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:48.361153 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b211d1-df7f-4127-b91d-4395564b4e56 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:48.361153 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b211d1-df7f-4127-b91d-4395564b4e56 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:48.362458 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 478/973] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:14:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:14:48.367583 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da585542-1c86-4f7f-85f5-75a13b2d482e None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:48.368065 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da585542-1c86-4f7f-85f5-75a13b2d482e None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:48.368244 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da585542-1c86-4f7f-85f5-75a13b2d482e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:48.388610 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-da585542-1c86-4f7f-85f5-75a13b2d482e None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:14:48.389892 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-da585542-1c86-4f7f-85f5-75a13b2d482e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:14:48.401134 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:48.472880 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 496/974] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:14:48 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:14:48.741603 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b211d1-df7f-4127-b91d-4395564b4e56 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:48.830351 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:48.841018 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b211d1-df7f-4127-b91d-4395564b4e56 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:48.873881 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85904000-9a4c-484f-b97d-e39a502ec783 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:48.874497 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85904000-9a4c-484f-b97d-e39a502ec783 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:48.874770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85904000-9a4c-484f-b97d-e39a502ec783 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:48.875037 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85904000-9a4c-484f-b97d-e39a502ec783 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:48.875612 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85904000-9a4c-484f-b97d-e39a502ec783 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:48.876959 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85904000-9a4c-484f-b97d-e39a502ec783 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:48.885432 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-85904000-9a4c-484f-b97d-e39a502ec783 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 27 01:14:48.885432 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:14:48.885432 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 27 01:14:48.885432 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 27 01:14:48.885432 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 27 01:14:48.885432 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 27 01:14:48.885432 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 27 01:14:48.885432 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:14:48.885432 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 27 01:14:48.885432 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:14:48.885432 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 27 01:14:48.885432 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:14:48.885432 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:14:48.885432 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:14:48.885432 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:14:48.885432 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:14:48.885432 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:14:48.887254 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:14:48.887254 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:14:48.887254 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:14:48.887254 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:14:48.887254 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:14:48.887254 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:14:48.887254 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 27 01:14:48.887254 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 27 01:14:48.887254 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 27 01:14:48.887254 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 27 01:14:48.887254 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:14:48.887254 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:14:48.887254 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:14:48.887254 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:14:48.887254 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:14:48.887254 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:14:48.888493 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:14:48.888493 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:14:48.888493 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:14:48.888493 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:14:48.888493 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:14:48.888493 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:14:48.888493 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:14:48.888493 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:14:48.888493 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:14:48.888493 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:14:48.888493 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 27 01:14:48.888493 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:14:48.888493 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 27 01:14:48.888493 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:14:48.888493 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 479/975] 10.222.0.42 () {56 vars in 1117 bytes} [Thu Oct 27 01:14:48 2022] GET /identity/v3/services/placement => generated 90 bytes in 145 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:14:48.893233 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da585542-1c86-4f7f-85f5-75a13b2d482e None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:48.988234 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da585542-1c86-4f7f-85f5-75a13b2d482e None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:49.021332 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecaff8c1-516c-431a-982c-e4ef909b037b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:49.021860 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecaff8c1-516c-431a-982c-e4ef909b037b None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:49.022135 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecaff8c1-516c-431a-982c-e4ef909b037b None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:49.022401 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecaff8c1-516c-431a-982c-e4ef909b037b None admin] PATH_INFO: `/v3/services` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:49.023021 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecaff8c1-516c-431a-982c-e4ef909b037b None admin] RBAC: Adding query filter params (name=placement) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:14:49.023315 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecaff8c1-516c-431a-982c-e4ef909b037b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:49.024663 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecaff8c1-516c-431a-982c-e4ef909b037b None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:49.030234 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 497/976] 10.222.0.42 () {56 vars in 1116 bytes} [Thu Oct 27 01:14:48 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:14:49.036110 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85904000-9a4c-484f-b97d-e39a502ec783 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:49.132177 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85904000-9a4c-484f-b97d-e39a502ec783 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:49.164889 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dbdb79f-efbb-437d-9a4f-bcef427742d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:49.165498 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbdb79f-efbb-437d-9a4f-bcef427742d2 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:49.165776 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbdb79f-efbb-437d-9a4f-bcef427742d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:49.166046 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbdb79f-efbb-437d-9a4f-bcef427742d2 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:49.166737 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dbdb79f-efbb-437d-9a4f-bcef427742d2 None admin] RBAC: Adding query filter params (interface=public, service_id=8ff76113e07841c9815cfc0b1390af6e, region_id=RegionOne) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:14:49.167042 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dbdb79f-efbb-437d-9a4f-bcef427742d2 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:49.168540 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dbdb79f-efbb-437d-9a4f-bcef427742d2 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:49.173324 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 480/977] 10.222.0.42 () {56 vars in 1251 bytes} [Thu Oct 27 01:14:49 2022] GET /identity/v3/endpoints?service_id=8ff76113e07841c9815cfc0b1390af6e&interface=public®ion_id=RegionOne => generated 194 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:14:49.179116 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecaff8c1-516c-431a-982c-e4ef909b037b None admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:49.182780 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:49.274962 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecaff8c1-516c-431a-982c-e4ef909b037b None admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:49.307736 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-862274e0-ede8-45aa-812d-f6054c58b0f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:49.308320 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862274e0-ede8-45aa-812d-f6054c58b0f0 None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:49.308620 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862274e0-ede8-45aa-812d-f6054c58b0f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:49.308884 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862274e0-ede8-45aa-812d-f6054c58b0f0 None admin] PATH_INFO: `/v3/services` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:49.309498 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-862274e0-ede8-45aa-812d-f6054c58b0f0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:49.310833 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-862274e0-ede8-45aa-812d-f6054c58b0f0 None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:49.317326 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 498/978] 10.222.0.42 () {56 vars in 1087 bytes} [Thu Oct 27 01:14:49 2022] GET /identity/v3/services => generated 2191 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:14:51.495370 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1888fc0b-4a5a-4af8-9d0c-f0f21cb42adc None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:51.496410 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1888fc0b-4a5a-4af8-9d0c-f0f21cb42adc None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:51.496410 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1888fc0b-4a5a-4af8-9d0c-f0f21cb42adc None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:51.497307 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 481/979] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:14:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:14:51.502542 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fab9f48-767b-4efb-ac7a-a81447c9f62a None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:51.502926 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fab9f48-767b-4efb-ac7a-a81447c9f62a None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:51.503092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fab9f48-767b-4efb-ac7a-a81447c9f62a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:51.521256 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-2fab9f48-767b-4efb-ac7a-a81447c9f62a None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:14:51.522426 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-2fab9f48-767b-4efb-ac7a-a81447c9f62a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:14:51.581313 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:51.586812 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 499/980] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:14:51 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 27 01:14:51.853317 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1888fc0b-4a5a-4af8-9d0c-f0f21cb42adc None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:51.873422 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:51.953622 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1888fc0b-4a5a-4af8-9d0c-f0f21cb42adc None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:51.985589 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82059724-410d-496d-b9b3-123535ac368e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:51.986200 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82059724-410d-496d-b9b3-123535ac368e None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:51.986478 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82059724-410d-496d-b9b3-123535ac368e None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:51.986750 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82059724-410d-496d-b9b3-123535ac368e None admin] PATH_INFO: `/v3/services/placement` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:51.987326 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82059724-410d-496d-b9b3-123535ac368e None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:51.988686 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82059724-410d-496d-b9b3-123535ac368e None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:51.997723 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-82059724-410d-496d-b9b3-123535ac368e None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 27 01:14:51.997723 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:14:51.997723 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 27 01:14:51.997723 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 27 01:14:51.997723 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 27 01:14:51.997723 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 27 01:14:51.997723 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 27 01:14:51.997723 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:14:51.997723 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 27 01:14:51.997723 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:14:51.997723 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 27 01:14:51.997723 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:14:51.997723 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 27 01:14:51.997723 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 27 01:14:51.997723 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 27 01:14:51.997723 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 27 01:14:51.997723 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 27 01:14:51.998904 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 27 01:14:51.998904 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 27 01:14:51.998904 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 27 01:14:51.998904 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 27 01:14:51.998904 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 27 01:14:51.998904 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 27 01:14:51.998904 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 27 01:14:51.998904 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 27 01:14:51.998904 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 27 01:14:51.998904 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 27 01:14:51.998904 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 27 01:14:51.998904 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 27 01:14:51.998904 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 27 01:14:51.998904 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 27 01:14:51.998904 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 27 01:14:51.998904 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 27 01:14:51.999944 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 27 01:14:51.999944 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 27 01:14:51.999944 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 27 01:14:51.999944 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 27 01:14:51.999944 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 27 01:14:51.999944 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 27 01:14:51.999944 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 27 01:14:51.999944 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 27 01:14:51.999944 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 27 01:14:51.999944 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 27 01:14:51.999944 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 27 01:14:51.999944 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 27 01:14:51.999944 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 27 01:14:51.999944 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 27 01:14:51.999944 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 482/981] 10.222.0.42 () {56 vars in 1117 bytes} [Thu Oct 27 01:14:51 2022] GET /identity/v3/services/placement => generated 90 bytes in 146 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 27 01:14:52.004853 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fab9f48-767b-4efb-ac7a-a81447c9f62a None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:52.098875 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fab9f48-767b-4efb-ac7a-a81447c9f62a None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:52.132790 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1072b606-09eb-4cbe-abc1-06da7799770f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:52.133406 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1072b606-09eb-4cbe-abc1-06da7799770f None admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:52.133689 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1072b606-09eb-4cbe-abc1-06da7799770f None admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:52.133961 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1072b606-09eb-4cbe-abc1-06da7799770f None admin] PATH_INFO: `/v3/services` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:52.134578 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1072b606-09eb-4cbe-abc1-06da7799770f None admin] RBAC: Adding query filter params (name=placement) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:14:52.134879 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1072b606-09eb-4cbe-abc1-06da7799770f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:52.136231 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1072b606-09eb-4cbe-abc1-06da7799770f None admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:52.141430 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 500/982] 10.222.0.42 () {56 vars in 1116 bytes} [Thu Oct 27 01:14:52 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:14:52.147009 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82059724-410d-496d-b9b3-123535ac368e None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:14:52.226906 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:14:52.245272 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82059724-410d-496d-b9b3-123535ac368e None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:14:52.279647 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-904482c8-5a25-4211-9950-3a67d398570c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:14:52.280242 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904482c8-5a25-4211-9950-3a67d398570c None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:14:52.280525 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904482c8-5a25-4211-9950-3a67d398570c None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:14:52.280799 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904482c8-5a25-4211-9950-3a67d398570c None admin] PATH_INFO: `/v3/endpoints` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:14:52.281500 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-904482c8-5a25-4211-9950-3a67d398570c None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:14:52.282863 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-904482c8-5a25-4211-9950-3a67d398570c None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:14:52.302025 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 483/983] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:14:52 2022] POST /identity/v3/endpoints => generated 331 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:15:02.138551 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=69 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:15:02.298188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=30,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:15:56.595029 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abca9d97-e59e-448d-bc99-07b237d10b71 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:15:56.595595 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abca9d97-e59e-448d-bc99-07b237d10b71 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:15:56.595595 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abca9d97-e59e-448d-bc99-07b237d10b71 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:15:56.600840 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 501/984] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:15:56 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:15:56.603222 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26c8347-064e-49db-aad1-ee9f7c107174 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:15:56.604093 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26c8347-064e-49db-aad1-ee9f7c107174 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:15:56.604535 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26c8347-064e-49db-aad1-ee9f7c107174 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:15:56.743501 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-f26c8347-064e-49db-aad1-ee9f7c107174 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:15:56.744937 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f26c8347-064e-49db-aad1-ee9f7c107174 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:15:57.022581 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 484/985] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:15:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 421 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:15:57.347958 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eaa4130-719b-4db0-b3db-22427bc8804b None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:15:57.348285 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eaa4130-719b-4db0-b3db-22427bc8804b None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:15:57.348503 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eaa4130-719b-4db0-b3db-22427bc8804b None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:15:57.350148 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 502/986] 10.222.0.42 () {54 vars in 965 bytes} [Thu Oct 27 01:15:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:15:57.357083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387f5346-8897-4d68-b7b9-6e2b5046b771 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:15:57.357374 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387f5346-8897-4d68-b7b9-6e2b5046b771 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:15:57.357724 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387f5346-8897-4d68-b7b9-6e2b5046b771 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:15:57.445265 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-387f5346-8897-4d68-b7b9-6e2b5046b771 None None] MFA Rules not processed for user `4072417990c94b0488871332d28224ba`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:15:57.446674 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-387f5346-8897-4d68-b7b9-6e2b5046b771 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:15:57.469832 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:15:57.553259 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eaa4130-719b-4db0-b3db-22427bc8804b None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:15:57.555484 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-2eaa4130-719b-4db0-b3db-22427bc8804b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:15:57.564790 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 485/987] 10.222.0.42 () {58 vars in 1062 bytes} [Thu Oct 27 01:15:57 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:15:57.678867 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eaa4130-719b-4db0-b3db-22427bc8804b None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:15:57.765685 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a58090e4-b563-4512-b8ed-ef9a1180c7e5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:15:57.766313 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58090e4-b563-4512-b8ed-ef9a1180c7e5 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:15:57.766591 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58090e4-b563-4512-b8ed-ef9a1180c7e5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:15:57.766866 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58090e4-b563-4512-b8ed-ef9a1180c7e5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:15:57.768566 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-a58090e4-b563-4512-b8ed-ef9a1180c7e5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:15:57.843679 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:15:57.908989 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a58090e4-b563-4512-b8ed-ef9a1180c7e5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:15:57.910541 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a58090e4-b563-4512-b8ed-ef9a1180c7e5 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:15:57.933361 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 503/988] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:15:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 27 01:16:00.418424 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51da76dd-3eaa-4dc4-b1a3-dbb3b6ffa69d None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:00.419003 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51da76dd-3eaa-4dc4-b1a3-dbb3b6ffa69d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:00.419003 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51da76dd-3eaa-4dc4-b1a3-dbb3b6ffa69d None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:00.420312 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 486/989] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:16:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:16:00.426076 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397d4ef4-a2bf-4adb-a087-9a209019abff None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:00.426597 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397d4ef4-a2bf-4adb-a087-9a209019abff None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:00.426597 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397d4ef4-a2bf-4adb-a087-9a209019abff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:00.498286 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-397d4ef4-a2bf-4adb-a087-9a209019abff None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:16:00.499480 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-397d4ef4-a2bf-4adb-a087-9a209019abff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:00.595950 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 504/990] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:16:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:16:00.878826 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51da76dd-3eaa-4dc4-b1a3-dbb3b6ffa69d None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:16:00.880353 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-51da76dd-3eaa-4dc4-b1a3-dbb3b6ffa69d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:00.981952 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51da76dd-3eaa-4dc4-b1a3-dbb3b6ffa69d None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:16:01.051963 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:16:01.054657 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0f8ed0-f90f-4ed8-9e68-a65cc7e3a65b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:16:01.055761 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0f8ed0-f90f-4ed8-9e68-a65cc7e3a65b service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:01.056228 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0f8ed0-f90f-4ed8-9e68-a65cc7e3a65b service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:01.056540 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0f8ed0-f90f-4ed8-9e68-a65cc7e3a65b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:01.058619 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-bc0f8ed0-f90f-4ed8-9e68-a65cc7e3a65b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:01.164980 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0f8ed0-f90f-4ed8-9e68-a65cc7e3a65b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:16:01.166656 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0f8ed0-f90f-4ed8-9e68-a65cc7e3a65b service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:16:01.184900 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 487/991] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:16:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:16:03.641649 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d027b2b-205a-41f3-ad17-1aad5da4e5f6 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:03.642180 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d027b2b-205a-41f3-ad17-1aad5da4e5f6 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:03.642180 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d027b2b-205a-41f3-ad17-1aad5da4e5f6 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:03.643540 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 505/992] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:16:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:16:03.649874 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77e35ef-89d0-44a6-922d-e12712ca528b None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:03.650278 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77e35ef-89d0-44a6-922d-e12712ca528b None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:03.650424 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77e35ef-89d0-44a6-922d-e12712ca528b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:03.716336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-e77e35ef-89d0-44a6-922d-e12712ca528b None None] MFA Rules not processed for user `4ed8fe9600cf453685d7c281db64fb87`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:16:03.717572 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e77e35ef-89d0-44a6-922d-e12712ca528b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:03.771906 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:16:03.817465 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 488/993] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:16:03 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:16:04.107880 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d027b2b-205a-41f3-ad17-1aad5da4e5f6 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:16:04.109429 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-9d027b2b-205a-41f3-ad17-1aad5da4e5f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:04.120516 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:16:04.213790 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d027b2b-205a-41f3-ad17-1aad5da4e5f6 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:16:04.286993 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e910c30-f39f-4518-b151-627431fc5276 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:16:04.287579 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e910c30-f39f-4518-b151-627431fc5276 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:04.287851 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e910c30-f39f-4518-b151-627431fc5276 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:04.288113 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e910c30-f39f-4518-b151-627431fc5276 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:04.290278 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-3e910c30-f39f-4518-b151-627431fc5276 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:04.350967 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:16:04.397342 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e910c30-f39f-4518-b151-627431fc5276 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:16:04.399787 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e910c30-f39f-4518-b151-627431fc5276 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:16:04.418708 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 506/994] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:16:04 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:16:07.139137 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205fda3e-9e3a-457a-a999-ea4ab045a815 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:07.139641 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205fda3e-9e3a-457a-a999-ea4ab045a815 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:07.139641 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205fda3e-9e3a-457a-a999-ea4ab045a815 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:07.140960 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 489/995] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:16:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:16:07.146123 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be33d49-f175-4bda-af31-da0fc97a9701 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:07.146534 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be33d49-f175-4bda-af31-da0fc97a9701 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:07.146666 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be33d49-f175-4bda-af31-da0fc97a9701 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:07.212190 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-2be33d49-f175-4bda-af31-da0fc97a9701 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:16:07.213419 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-2be33d49-f175-4bda-af31-da0fc97a9701 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:07.306711 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 507/996] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:16:07 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:16:07.598979 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-205fda3e-9e3a-457a-a999-ea4ab045a815 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:16:07.600503 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-205fda3e-9e3a-457a-a999-ea4ab045a815 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:07.703495 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-205fda3e-9e3a-457a-a999-ea4ab045a815 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:16:07.775056 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0867a9-9018-4d13-a347-8f59db14bf5f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:16:07.775651 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0867a9-9018-4d13-a347-8f59db14bf5f service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:07.775929 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0867a9-9018-4d13-a347-8f59db14bf5f service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:07.776199 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0867a9-9018-4d13-a347-8f59db14bf5f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:07.777818 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ec0867a9-9018-4d13-a347-8f59db14bf5f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:07.785129 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:16:07.886993 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0867a9-9018-4d13-a347-8f59db14bf5f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:16:07.888353 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0867a9-9018-4d13-a347-8f59db14bf5f service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:16:07.906405 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 490/997] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:16:07 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:16:10.759087 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fc511a-552c-4080-b6d0-50dbd7a6f470 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:10.759654 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fc511a-552c-4080-b6d0-50dbd7a6f470 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:10.759654 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fc511a-552c-4080-b6d0-50dbd7a6f470 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:10.760965 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 508/998] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:16:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:16:10.766767 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088f4492-9ded-48b3-a61f-362536369502 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:10.767311 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088f4492-9ded-48b3-a61f-362536369502 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:10.767311 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088f4492-9ded-48b3-a61f-362536369502 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:10.845698 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-088f4492-9ded-48b3-a61f-362536369502 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:16:10.846981 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-088f4492-9ded-48b3-a61f-362536369502 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:10.925205 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:16:10.974651 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 491/999] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:16:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:16:11.292660 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631ef67e-4363-4812-a48c-32cf805e9ca4 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:11.292940 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631ef67e-4363-4812-a48c-32cf805e9ca4 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:11.293238 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631ef67e-4363-4812-a48c-32cf805e9ca4 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:11.294558 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 509/1000] 10.222.0.42 () {54 vars in 965 bytes} [Thu Oct 27 01:16:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:16:11.300615 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b37567c-57bc-4b18-9505-9a5b3d7f8239 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:11.301209 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b37567c-57bc-4b18-9505-9a5b3d7f8239 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:11.301694 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b37567c-57bc-4b18-9505-9a5b3d7f8239 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:11.397743 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-9b37567c-57bc-4b18-9505-9a5b3d7f8239 None None] MFA Rules not processed for user `4072417990c94b0488871332d28224ba`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:16:11.399933 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-9b37567c-57bc-4b18-9505-9a5b3d7f8239 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:11.482295 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 492/1001] 10.222.0.42 () {58 vars in 1062 bytes} [Thu Oct 27 01:16:11 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:16:11.492323 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-631ef67e-4363-4812-a48c-32cf805e9ca4 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:16:11.493661 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-631ef67e-4363-4812-a48c-32cf805e9ca4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:11.516852 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:16:11.616065 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-631ef67e-4363-4812-a48c-32cf805e9ca4 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:16:11.689357 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca75bd5f-66db-4b24-bfd1-d5f11e1764c3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:16:11.690063 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca75bd5f-66db-4b24-bfd1-d5f11e1764c3 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:11.690343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca75bd5f-66db-4b24-bfd1-d5f11e1764c3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:11.690612 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca75bd5f-66db-4b24-bfd1-d5f11e1764c3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:11.692193 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-ca75bd5f-66db-4b24-bfd1-d5f11e1764c3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:11.767990 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:16:11.817622 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca75bd5f-66db-4b24-bfd1-d5f11e1764c3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:16:11.819063 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca75bd5f-66db-4b24-bfd1-d5f11e1764c3 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:16:11.838141 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 510/1002] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:16:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:16:14.311282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1548f0f-b500-442d-b62e-129a932cc9d0 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:14.311816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1548f0f-b500-442d-b62e-129a932cc9d0 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:14.311816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1548f0f-b500-442d-b62e-129a932cc9d0 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:14.313189 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 493/1003] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:16:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:16:14.318384 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d663af-1974-499c-b2a1-2c6acb3e542b None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:14.318807 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d663af-1974-499c-b2a1-2c6acb3e542b None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:14.318935 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d663af-1974-499c-b2a1-2c6acb3e542b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:14.386011 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-39d663af-1974-499c-b2a1-2c6acb3e542b None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:16:14.387190 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-39d663af-1974-499c-b2a1-2c6acb3e542b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:14.482853 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 511/1004] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:16:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:16:14.763435 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1548f0f-b500-442d-b62e-129a932cc9d0 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:16:14.765009 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e1548f0f-b500-442d-b62e-129a932cc9d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:14.829302 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:16:14.876241 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1548f0f-b500-442d-b62e-129a932cc9d0 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:16:14.952554 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe10bda-0554-4084-bc84-b112cbc10d6c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:16:14.953082 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe10bda-0554-4084-bc84-b112cbc10d6c service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:14.953367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe10bda-0554-4084-bc84-b112cbc10d6c service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:14.953631 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe10bda-0554-4084-bc84-b112cbc10d6c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:14.955192 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-afe10bda-0554-4084-bc84-b112cbc10d6c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:15.072215 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:16:15.074721 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afe10bda-0554-4084-bc84-b112cbc10d6c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:16:15.076586 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afe10bda-0554-4084-bc84-b112cbc10d6c service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:16:15.096877 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 494/1005] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:16:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:16:17.333283 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797d9929-645a-47e8-9aa5-783949b86e84 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:17.333714 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797d9929-645a-47e8-9aa5-783949b86e84 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:17.333930 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797d9929-645a-47e8-9aa5-783949b86e84 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:17.335247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 512/1006] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:16:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:16:17.340868 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3df03be-85fc-410c-80b3-378365893bff None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:17.341347 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3df03be-85fc-410c-80b3-378365893bff None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:17.341460 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3df03be-85fc-410c-80b3-378365893bff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:17.409149 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-e3df03be-85fc-410c-80b3-378365893bff None None] MFA Rules not processed for user `4ed8fe9600cf453685d7c281db64fb87`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:16:17.410281 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e3df03be-85fc-410c-80b3-378365893bff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:17.508134 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 495/1007] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:16:17 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:16:17.793237 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797d9929-645a-47e8-9aa5-783949b86e84 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:16:17.794857 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-797d9929-645a-47e8-9aa5-783949b86e84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:17.835012 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:16:17.910591 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797d9929-645a-47e8-9aa5-783949b86e84 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:16:17.986385 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34e8372-54ee-401d-bf0d-0b95f8c4abf6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:16:17.987011 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34e8372-54ee-401d-bf0d-0b95f8c4abf6 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:17.987315 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34e8372-54ee-401d-bf0d-0b95f8c4abf6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:17.987585 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34e8372-54ee-401d-bf0d-0b95f8c4abf6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:17.989207 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-c34e8372-54ee-401d-bf0d-0b95f8c4abf6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:18.072674 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:16:18.107655 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c34e8372-54ee-401d-bf0d-0b95f8c4abf6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:16:18.109127 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c34e8372-54ee-401d-bf0d-0b95f8c4abf6 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:16:18.128805 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 513/1008] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:16:17 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:16:20.528288 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afdb632e-b454-45d0-b09c-ecfb53a8fd3d None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:20.528822 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afdb632e-b454-45d0-b09c-ecfb53a8fd3d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:20.528822 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afdb632e-b454-45d0-b09c-ecfb53a8fd3d None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:20.530191 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 496/1009] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:16:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 27 01:16:20.535572 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e9af50-063a-4e75-88f0-9992b723ff98 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:20.535969 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e9af50-063a-4e75-88f0-9992b723ff98 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:20.536124 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e9af50-063a-4e75-88f0-9992b723ff98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:20.605937 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-12e9af50-063a-4e75-88f0-9992b723ff98 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:16:20.607151 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-12e9af50-063a-4e75-88f0-9992b723ff98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:20.701887 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 514/1010] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:16:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:16:20.993290 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afdb632e-b454-45d0-b09c-ecfb53a8fd3d None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:16:20.994812 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-afdb632e-b454-45d0-b09c-ecfb53a8fd3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:21.064249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:16:21.098913 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afdb632e-b454-45d0-b09c-ecfb53a8fd3d None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:16:21.170595 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-353ec0d3-678e-473f-b7e3-fa470943981f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:16:21.171188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353ec0d3-678e-473f-b7e3-fa470943981f service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:21.171464 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353ec0d3-678e-473f-b7e3-fa470943981f service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:21.171730 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353ec0d3-678e-473f-b7e3-fa470943981f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:21.173327 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-353ec0d3-678e-473f-b7e3-fa470943981f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:21.280730 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-353ec0d3-678e-473f-b7e3-fa470943981f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:16:21.282195 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-353ec0d3-678e-473f-b7e3-fa470943981f service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:16:21.291979 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:16:21.300174 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 497/1011] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:16:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:16:23.510563 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df71633-cd90-4846-aa6e-5bfb5ad80e2f None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:23.511130 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df71633-cd90-4846-aa6e-5bfb5ad80e2f None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:23.511130 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df71633-cd90-4846-aa6e-5bfb5ad80e2f None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:23.512479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 515/1012] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:16:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:16:23.517804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196dbfed-9c1d-4792-9ced-0d0a60dec465 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:23.518230 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196dbfed-9c1d-4792-9ced-0d0a60dec465 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:23.518358 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196dbfed-9c1d-4792-9ced-0d0a60dec465 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:23.583933 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-196dbfed-9c1d-4792-9ced-0d0a60dec465 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:16:23.585238 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-196dbfed-9c1d-4792-9ced-0d0a60dec465 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:23.681770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 498/1013] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:16:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:16:23.966897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df71633-cd90-4846-aa6e-5bfb5ad80e2f None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:16:23.968420 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-1df71633-cd90-4846-aa6e-5bfb5ad80e2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:24.008682 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:16:24.072070 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df71633-cd90-4846-aa6e-5bfb5ad80e2f None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:16:24.143448 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff87165-a5d0-4bc3-bb82-cbb8281f6ac7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:16:24.144041 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff87165-a5d0-4bc3-bb82-cbb8281f6ac7 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:24.144319 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff87165-a5d0-4bc3-bb82-cbb8281f6ac7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:24.144588 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff87165-a5d0-4bc3-bb82-cbb8281f6ac7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:24.146201 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-7ff87165-a5d0-4bc3-bb82-cbb8281f6ac7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:24.228748 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:16:24.254397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff87165-a5d0-4bc3-bb82-cbb8281f6ac7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:16:24.255816 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff87165-a5d0-4bc3-bb82-cbb8281f6ac7 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:16:24.274075 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 516/1014] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:16:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:16:26.779374 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bb0c1e-0255-4845-a53b-ce7f9a626899 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:26.779903 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bb0c1e-0255-4845-a53b-ce7f9a626899 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:26.779903 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bb0c1e-0255-4845-a53b-ce7f9a626899 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:26.781291 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 499/1015] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:16:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:16:26.786510 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4662068-9f98-4bd7-9d5e-fdb400f64b4e None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:26.786948 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4662068-9f98-4bd7-9d5e-fdb400f64b4e None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:26.787057 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4662068-9f98-4bd7-9d5e-fdb400f64b4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:26.857439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-d4662068-9f98-4bd7-9d5e-fdb400f64b4e None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:16:26.858662 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-d4662068-9f98-4bd7-9d5e-fdb400f64b4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:26.957835 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 517/1016] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:16:26 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:16:27.246979 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84bb0c1e-0255-4845-a53b-ce7f9a626899 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:16:27.248488 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-84bb0c1e-0255-4845-a53b-ce7f9a626899 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:27.322927 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:16:27.350178 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84bb0c1e-0255-4845-a53b-ce7f9a626899 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:16:27.422346 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eedb81ae-2eab-4fdd-a4ff-1589a887c1b1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:16:27.422945 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedb81ae-2eab-4fdd-a4ff-1589a887c1b1 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:27.423223 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedb81ae-2eab-4fdd-a4ff-1589a887c1b1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:27.423490 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedb81ae-2eab-4fdd-a4ff-1589a887c1b1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:27.425073 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-eedb81ae-2eab-4fdd-a4ff-1589a887c1b1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:27.531907 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eedb81ae-2eab-4fdd-a4ff-1589a887c1b1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:16:27.533276 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eedb81ae-2eab-4fdd-a4ff-1589a887c1b1 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:16:27.550561 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 500/1017] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:16:27 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:16:31.620153 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe693aa3-3999-4bc5-93b4-738cfbb9e385 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:31.620688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe693aa3-3999-4bc5-93b4-738cfbb9e385 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:31.620688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe693aa3-3999-4bc5-93b4-738cfbb9e385 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:31.622014 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 518/1018] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:16:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:16:31.627158 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72557ac5-eb9a-4c4c-a224-e5c6594eda11 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:31.627571 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72557ac5-eb9a-4c4c-a224-e5c6594eda11 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:31.627703 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72557ac5-eb9a-4c4c-a224-e5c6594eda11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:31.633246 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:16:31.696595 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-72557ac5-eb9a-4c4c-a224-e5c6594eda11 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:16:31.697829 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-72557ac5-eb9a-4c4c-a224-e5c6594eda11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:31.793334 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 501/1019] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:16:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:16:32.080146 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe693aa3-3999-4bc5-93b4-738cfbb9e385 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:16:32.081851 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-fe693aa3-3999-4bc5-93b4-738cfbb9e385 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:32.131025 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:16:32.186192 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe693aa3-3999-4bc5-93b4-738cfbb9e385 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:16:32.260949 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-103431b5-8bdf-4fa3-9d2b-8622d72b39a7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:16:32.261584 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103431b5-8bdf-4fa3-9d2b-8622d72b39a7 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:32.261862 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103431b5-8bdf-4fa3-9d2b-8622d72b39a7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:32.262132 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103431b5-8bdf-4fa3-9d2b-8622d72b39a7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:32.263721 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-103431b5-8bdf-4fa3-9d2b-8622d72b39a7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:32.358029 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:16:32.374431 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-103431b5-8bdf-4fa3-9d2b-8622d72b39a7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:16:32.375970 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-103431b5-8bdf-4fa3-9d2b-8622d72b39a7 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:16:32.395196 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 519/1020] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:16:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:16:35.219453 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f188f987-6dd7-4830-8403-fc3ac2059280 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:35.220038 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f188f987-6dd7-4830-8403-fc3ac2059280 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:35.220038 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f188f987-6dd7-4830-8403-fc3ac2059280 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:35.221364 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 502/1021] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:16:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:16:35.227485 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b2ee3b-2828-4873-a578-053d8cb8045b None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:35.227924 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b2ee3b-2828-4873-a578-053d8cb8045b None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:35.228078 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b2ee3b-2828-4873-a578-053d8cb8045b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:35.294041 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-12b2ee3b-2828-4873-a578-053d8cb8045b None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:16:35.295245 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-12b2ee3b-2828-4873-a578-053d8cb8045b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:35.396227 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 520/1022] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:16:35 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:16:35.681691 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f188f987-6dd7-4830-8403-fc3ac2059280 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:16:35.683209 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f188f987-6dd7-4830-8403-fc3ac2059280 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:35.776879 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:16:35.795478 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f188f987-6dd7-4830-8403-fc3ac2059280 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:16:35.872959 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453d4967-74aa-468e-8b22-b9ea1014cc60 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:16:35.873503 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453d4967-74aa-468e-8b22-b9ea1014cc60 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:35.873779 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453d4967-74aa-468e-8b22-b9ea1014cc60 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:35.874041 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453d4967-74aa-468e-8b22-b9ea1014cc60 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:35.875606 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-453d4967-74aa-468e-8b22-b9ea1014cc60 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:35.985916 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-453d4967-74aa-468e-8b22-b9ea1014cc60 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:16:35.987354 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-453d4967-74aa-468e-8b22-b9ea1014cc60 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:16:36.004739 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 503/1023] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:16:35 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:16:40.171941 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066bb2e0-f97a-4114-9f5d-a188a200d814 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:40.172448 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066bb2e0-f97a-4114-9f5d-a188a200d814 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:40.172575 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066bb2e0-f97a-4114-9f5d-a188a200d814 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:40.173856 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 521/1024] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:16:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:16:40.179752 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bed471a-2fb6-4099-80bb-cf32219115a9 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:40.180197 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bed471a-2fb6-4099-80bb-cf32219115a9 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:40.180297 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bed471a-2fb6-4099-80bb-cf32219115a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:40.196926 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:16:40.250198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-8bed471a-2fb6-4099-80bb-cf32219115a9 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:16:40.251414 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-8bed471a-2fb6-4099-80bb-cf32219115a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:40.350612 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 504/1025] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:16:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:16:40.636369 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-066bb2e0-f97a-4114-9f5d-a188a200d814 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:16:40.637965 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-066bb2e0-f97a-4114-9f5d-a188a200d814 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:40.695372 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:16:40.743620 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-066bb2e0-f97a-4114-9f5d-a188a200d814 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:16:40.821708 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef5cbad-4bb0-4bbb-bddf-b3d3609bbbb1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:16:40.822310 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef5cbad-4bb0-4bbb-bddf-b3d3609bbbb1 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:40.822583 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef5cbad-4bb0-4bbb-bddf-b3d3609bbbb1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:40.822848 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef5cbad-4bb0-4bbb-bddf-b3d3609bbbb1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:40.824477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-4ef5cbad-4bb0-4bbb-bddf-b3d3609bbbb1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:40.936369 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:16:40.941765 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef5cbad-4bb0-4bbb-bddf-b3d3609bbbb1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:16:40.943633 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef5cbad-4bb0-4bbb-bddf-b3d3609bbbb1 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:16:40.965412 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 522/1026] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:16:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:16:46.168625 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ec65ce-80c3-4201-bbcb-79a71fb81a39 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:46.168625 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ec65ce-80c3-4201-bbcb-79a71fb81a39 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:46.169161 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ec65ce-80c3-4201-bbcb-79a71fb81a39 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:46.170733 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 505/1027] 10.222.0.42 () {54 vars in 921 bytes} [Thu Oct 27 01:16:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:16:46.181093 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058f9354-e12a-4e8e-95c2-6bb6daf664c3 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:46.181093 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058f9354-e12a-4e8e-95c2-6bb6daf664c3 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:46.181093 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058f9354-e12a-4e8e-95c2-6bb6daf664c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:46.287688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-058f9354-e12a-4e8e-95c2-6bb6daf664c3 None None] MFA Rules not processed for user `62051562fc0b4cd08750d2293e71787c`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:16:46.288610 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-058f9354-e12a-4e8e-95c2-6bb6daf664c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:46.463156 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 523/1028] 10.222.0.42 () {58 vars in 1018 bytes} [Thu Oct 27 01:16:46 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 288 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:16:50.348441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=59 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:16:50.544351 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c83e501-654e-4773-8eb9-61e4a14292f5 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:50.544818 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c83e501-654e-4773-8eb9-61e4a14292f5 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:50.545312 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c83e501-654e-4773-8eb9-61e4a14292f5 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:50.547107 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 506/1029] 10.222.0.42 () {54 vars in 921 bytes} [Thu Oct 27 01:16:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:16:50.552927 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4670a7c-0fcc-4de3-82ee-67bf6bbe3172 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:50.553427 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4670a7c-0fcc-4de3-82ee-67bf6bbe3172 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:50.553656 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4670a7c-0fcc-4de3-82ee-67bf6bbe3172 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:50.607660 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:16:50.631955 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-c4670a7c-0fcc-4de3-82ee-67bf6bbe3172 None None] MFA Rules not processed for user `62051562fc0b4cd08750d2293e71787c`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:16:50.633255 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-c4670a7c-0fcc-4de3-82ee-67bf6bbe3172 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:50.746472 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 524/1030] 10.222.0.42 () {58 vars in 1018 bytes} [Thu Oct 27 01:16:50 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:16:52.254789 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591bebe4-2f0d-40cd-aebe-3741bb97f9d9 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:52.255380 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591bebe4-2f0d-40cd-aebe-3741bb97f9d9 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:52.255894 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591bebe4-2f0d-40cd-aebe-3741bb97f9d9 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:52.257344 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 507/1031] 10.222.0.42 () {54 vars in 921 bytes} [Thu Oct 27 01:16:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:16:52.262419 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d687ee62-4c01-45ce-8f6e-c0ec4ff37b4f None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:52.262822 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d687ee62-4c01-45ce-8f6e-c0ec4ff37b4f None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:52.263103 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d687ee62-4c01-45ce-8f6e-c0ec4ff37b4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:52.342001 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-d687ee62-4c01-45ce-8f6e-c0ec4ff37b4f None None] MFA Rules not processed for user `62051562fc0b4cd08750d2293e71787c`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:16:52.343219 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-d687ee62-4c01-45ce-8f6e-c0ec4ff37b4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:52.384843 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:16:52.449743 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 525/1032] 10.222.0.42 () {58 vars in 1018 bytes} [Thu Oct 27 01:16:52 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:16:56.026839 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f19660-eb0e-4cfb-80cd-4f6bf1e06e06 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:56.027353 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f19660-eb0e-4cfb-80cd-4f6bf1e06e06 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:56.027353 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f19660-eb0e-4cfb-80cd-4f6bf1e06e06 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:56.028749 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 508/1033] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:16:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:16:56.034361 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b871e0e-6919-43fb-acec-804424b99eb9 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:56.034808 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b871e0e-6919-43fb-acec-804424b99eb9 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:56.034908 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b871e0e-6919-43fb-acec-804424b99eb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:56.118705 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-6b871e0e-6919-43fb-acec-804424b99eb9 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:16:56.120047 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-6b871e0e-6919-43fb-acec-804424b99eb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:56.225104 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 526/1034] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:16:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:16:56.524432 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb9be58-0124-431a-9a3d-81c3ab1d53ba None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:56.524975 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb9be58-0124-431a-9a3d-81c3ab1d53ba None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:56.525369 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb9be58-0124-431a-9a3d-81c3ab1d53ba None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:56.527195 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 509/1035] 10.222.0.42 () {54 vars in 961 bytes} [Thu Oct 27 01:16:56 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:16:56.535670 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4449201a-8ee3-4f57-9e7d-eb0996c99f69 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:56.536969 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4449201a-8ee3-4f57-9e7d-eb0996c99f69 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:56.537313 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4449201a-8ee3-4f57-9e7d-eb0996c99f69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:56.558952 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:16:56.629745 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-4449201a-8ee3-4f57-9e7d-eb0996c99f69 None None] MFA Rules not processed for user `51033d56f85f4282a49ded6338440677`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:16:56.631189 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-4449201a-8ee3-4f57-9e7d-eb0996c99f69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:56.758723 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 527/1036] 10.222.0.42 () {58 vars in 1058 bytes} [Thu Oct 27 01:16:56 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:16:56.766457 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb9be58-0124-431a-9a3d-81c3ab1d53ba None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:16:56.767964 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-cdb9be58-0124-431a-9a3d-81c3ab1d53ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:56.940732 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb9be58-0124-431a-9a3d-81c3ab1d53ba None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:16:57.019625 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:16:57.065489 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99237047-3f6a-492d-967c-aa07582e3cbd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:16:57.066274 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99237047-3f6a-492d-967c-aa07582e3cbd service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:57.066452 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99237047-3f6a-492d-967c-aa07582e3cbd service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:57.066722 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99237047-3f6a-492d-967c-aa07582e3cbd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:57.068730 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-99237047-3f6a-492d-967c-aa07582e3cbd service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:57.169050 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99237047-3f6a-492d-967c-aa07582e3cbd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:16:57.170669 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99237047-3f6a-492d-967c-aa07582e3cbd service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:16:57.191056 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 510/1037] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:16:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 425 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 27 01:16:59.605448 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66967d9a-efdd-48d0-a847-fd55bee94466 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:59.605955 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66967d9a-efdd-48d0-a847-fd55bee94466 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:59.605955 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66967d9a-efdd-48d0-a847-fd55bee94466 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:59.611190 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 528/1038] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:16:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:16:59.619490 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12528253-9d2f-4d7f-a63b-9bc6fe9654e0 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:16:59.620030 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12528253-9d2f-4d7f-a63b-9bc6fe9654e0 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:16:59.621267 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12528253-9d2f-4d7f-a63b-9bc6fe9654e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:16:59.702707 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-12528253-9d2f-4d7f-a63b-9bc6fe9654e0 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:16:59.703978 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-12528253-9d2f-4d7f-a63b-9bc6fe9654e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:16:59.743987 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:16:59.810233 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 511/1039] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:16:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:17:00.105480 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66967d9a-efdd-48d0-a847-fd55bee94466 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:17:00.107135 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-66967d9a-efdd-48d0-a847-fd55bee94466 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:00.193120 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:00.308412 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66967d9a-efdd-48d0-a847-fd55bee94466 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:17:00.440578 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a2385c-ee6b-4796-91d1-9cbda3445786 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:17:00.441258 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a2385c-ee6b-4796-91d1-9cbda3445786 service nova] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:00.441537 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a2385c-ee6b-4796-91d1-9cbda3445786 service nova] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:00.441809 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a2385c-ee6b-4796-91d1-9cbda3445786 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:00.443435 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-d0a2385c-ee6b-4796-91d1-9cbda3445786 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:00.466787 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:00.537217 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a2385c-ee6b-4796-91d1-9cbda3445786 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:17:00.538803 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a2385c-ee6b-4796-91d1-9cbda3445786 service nova] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:17:00.558641 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 529/1040] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:17:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 454 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:17:03.111104 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8aac77a-052e-4631-970e-6c844e4a030d None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:03.111612 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8aac77a-052e-4631-970e-6c844e4a030d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:03.111612 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8aac77a-052e-4631-970e-6c844e4a030d None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:03.112877 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 512/1041] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:17:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:17:03.118523 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74aba4ab-c82f-43e6-8138-749564df3310 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:03.118946 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74aba4ab-c82f-43e6-8138-749564df3310 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:03.119041 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74aba4ab-c82f-43e6-8138-749564df3310 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:03.190095 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-74aba4ab-c82f-43e6-8138-749564df3310 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:17:03.191284 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-74aba4ab-c82f-43e6-8138-749564df3310 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:03.293474 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:03.297165 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 530/1042] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:17:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:17:03.590916 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72362ed7-20aa-4ba0-aed1-69e54c72c231 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:03.591200 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72362ed7-20aa-4ba0-aed1-69e54c72c231 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:03.591644 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72362ed7-20aa-4ba0-aed1-69e54c72c231 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:03.593173 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 513/1043] 10.222.0.42 () {54 vars in 961 bytes} [Thu Oct 27 01:17:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:17:03.599499 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e1b222-fcd1-4b91-8f1b-83fa84ca2253 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:03.599780 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e1b222-fcd1-4b91-8f1b-83fa84ca2253 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:03.600052 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e1b222-fcd1-4b91-8f1b-83fa84ca2253 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:03.679867 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-92e1b222-fcd1-4b91-8f1b-83fa84ca2253 None None] MFA Rules not processed for user `51033d56f85f4282a49ded6338440677`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:17:03.681090 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-92e1b222-fcd1-4b91-8f1b-83fa84ca2253 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:03.807019 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 531/1044] 10.222.0.42 () {58 vars in 1058 bytes} [Thu Oct 27 01:17:03 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:17:03.834328 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72362ed7-20aa-4ba0-aed1-69e54c72c231 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:17:03.836149 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-72362ed7-20aa-4ba0-aed1-69e54c72c231 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:03.997422 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72362ed7-20aa-4ba0-aed1-69e54c72c231 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:17:04.018732 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:04.129766 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d9ccba-b9bc-4d48-853f-9a9d94acbb0a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:17:04.130544 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d9ccba-b9bc-4d48-853f-9a9d94acbb0a service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:04.130810 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d9ccba-b9bc-4d48-853f-9a9d94acbb0a service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:04.131090 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d9ccba-b9bc-4d48-853f-9a9d94acbb0a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:04.132841 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-64d9ccba-b9bc-4d48-853f-9a9d94acbb0a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:04.236352 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64d9ccba-b9bc-4d48-853f-9a9d94acbb0a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:17:04.238136 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64d9ccba-b9bc-4d48-853f-9a9d94acbb0a service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:17:04.258570 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 514/1045] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:17:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 425 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:17:04.301839 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376eccae-f7b6-4e81-8025-e34d54e76a96 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:04.302491 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376eccae-f7b6-4e81-8025-e34d54e76a96 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:04.302491 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376eccae-f7b6-4e81-8025-e34d54e76a96 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:04.303906 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 532/1046] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:17:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:17:04.312704 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbbc739d-ba85-46fa-96f7-ef983a330300 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:04.313125 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbbc739d-ba85-46fa-96f7-ef983a330300 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:04.313496 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbbc739d-ba85-46fa-96f7-ef983a330300 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:04.350674 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:04.396306 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-cbbc739d-ba85-46fa-96f7-ef983a330300 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:17:04.397650 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-cbbc739d-ba85-46fa-96f7-ef983a330300 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:04.522457 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 515/1047] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:17:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:17:04.860340 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff8df63-6b43-464d-a450-ce619d0ea465 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:04.860340 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff8df63-6b43-464d-a450-ce619d0ea465 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:04.860340 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff8df63-6b43-464d-a450-ce619d0ea465 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:04.862778 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 533/1048] 10.222.0.42 () {54 vars in 963 bytes} [Thu Oct 27 01:17:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:17:04.867061 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d84954c-0598-4c02-8643-0fd47adcafb9 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:04.867246 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d84954c-0598-4c02-8643-0fd47adcafb9 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:04.867409 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d84954c-0598-4c02-8643-0fd47adcafb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:04.946608 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-9d84954c-0598-4c02-8643-0fd47adcafb9 None None] MFA Rules not processed for user `d35a3d40a1924064859562a72bab0f6d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:17:04.947816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-9d84954c-0598-4c02-8643-0fd47adcafb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:05.000627 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:05.032947 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 516/1049] 10.222.0.42 () {58 vars in 1060 bytes} [Thu Oct 27 01:17:04 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:17:05.040799 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff8df63-6b43-464d-a450-ce619d0ea465 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:17:05.042454 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-fff8df63-6b43-464d-a450-ce619d0ea465 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:05.101004 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:05.159282 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff8df63-6b43-464d-a450-ce619d0ea465 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:17:05.247636 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea0ff33-70cf-4e0b-9bbe-e4831e38b439 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:17:05.248255 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea0ff33-70cf-4e0b-9bbe-e4831e38b439 service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:05.248525 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea0ff33-70cf-4e0b-9bbe-e4831e38b439 service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:05.248787 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea0ff33-70cf-4e0b-9bbe-e4831e38b439 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:05.250441 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-2ea0ff33-70cf-4e0b-9bbe-e4831e38b439 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:05.397807 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:05.423318 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea0ff33-70cf-4e0b-9bbe-e4831e38b439 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:17:05.426481 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea0ff33-70cf-4e0b-9bbe-e4831e38b439 service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:17:05.458686 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 534/1050] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:17:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 418 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:17:06.527014 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f05b6f-c43c-4271-ad90-4cce43bc144f None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:06.527579 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f05b6f-c43c-4271-ad90-4cce43bc144f None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:06.527579 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f05b6f-c43c-4271-ad90-4cce43bc144f None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:06.528961 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 517/1051] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:17:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:17:06.534427 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deaf3e11-eb6d-4a9d-9e57-ebbc044609ae None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:06.534850 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deaf3e11-eb6d-4a9d-9e57-ebbc044609ae None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:06.535117 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deaf3e11-eb6d-4a9d-9e57-ebbc044609ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:06.602767 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-deaf3e11-eb6d-4a9d-9e57-ebbc044609ae None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:17:06.603988 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-deaf3e11-eb6d-4a9d-9e57-ebbc044609ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:06.699273 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 535/1052] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:17:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:17:06.990867 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f05b6f-c43c-4271-ad90-4cce43bc144f None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:17:06.992533 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-99f05b6f-c43c-4271-ad90-4cce43bc144f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:07.156560 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f05b6f-c43c-4271-ad90-4cce43bc144f None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:17:07.209460 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:07.274805 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce37847-a643-4fd6-a9b5-bf6c0ef95c8c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:17:07.275524 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce37847-a643-4fd6-a9b5-bf6c0ef95c8c service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:07.275806 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce37847-a643-4fd6-a9b5-bf6c0ef95c8c service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:07.276080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce37847-a643-4fd6-a9b5-bf6c0ef95c8c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:07.277755 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-8ce37847-a643-4fd6-a9b5-bf6c0ef95c8c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:07.368921 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce37847-a643-4fd6-a9b5-bf6c0ef95c8c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:17:07.370577 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce37847-a643-4fd6-a9b5-bf6c0ef95c8c service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:17:07.391031 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 518/1053] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:17:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 401 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:17:09.636677 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b3836d-f06b-4435-9bb3-3306a753569a None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:09.637261 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b3836d-f06b-4435-9bb3-3306a753569a None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:09.637515 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b3836d-f06b-4435-9bb3-3306a753569a None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:09.639122 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 536/1054] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:17:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:17:09.644069 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c783af7-7c95-4f01-80b4-23d5df53853c None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:09.644492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c783af7-7c95-4f01-80b4-23d5df53853c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:09.645434 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c783af7-7c95-4f01-80b4-23d5df53853c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:09.714203 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-0c783af7-7c95-4f01-80b4-23d5df53853c None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:17:09.715398 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0c783af7-7c95-4f01-80b4-23d5df53853c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:09.722640 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:09.829604 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 519/1055] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:17:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:17:10.115251 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b3836d-f06b-4435-9bb3-3306a753569a None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:17:10.116899 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-e4b3836d-f06b-4435-9bb3-3306a753569a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:10.187053 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:10.279853 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b3836d-f06b-4435-9bb3-3306a753569a None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:17:10.406159 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33146650-8d23-41d2-9af4-837436aa049a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:17:10.406878 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33146650-8d23-41d2-9af4-837436aa049a service nova] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:10.407264 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33146650-8d23-41d2-9af4-837436aa049a service nova] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:10.407651 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33146650-8d23-41d2-9af4-837436aa049a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:10.409366 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-33146650-8d23-41d2-9af4-837436aa049a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:10.430808 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:10.504556 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33146650-8d23-41d2-9af4-837436aa049a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:17:10.506213 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33146650-8d23-41d2-9af4-837436aa049a service nova] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:17:10.525164 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 537/1056] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:17:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:17:12.783231 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b880fe4-1cdb-4f87-b660-bc2b147cca53 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:12.783758 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b880fe4-1cdb-4f87-b660-bc2b147cca53 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:12.783758 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b880fe4-1cdb-4f87-b660-bc2b147cca53 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:12.785852 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 520/1057] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:17:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:17:12.791279 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d527fa3b-60d6-45b1-96b1-f350ba3a4b37 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:12.791720 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d527fa3b-60d6-45b1-96b1-f350ba3a4b37 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:12.791818 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d527fa3b-60d6-45b1-96b1-f350ba3a4b37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:12.866502 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-d527fa3b-60d6-45b1-96b1-f350ba3a4b37 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:17:12.867718 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-d527fa3b-60d6-45b1-96b1-f350ba3a4b37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:12.985808 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:12.994740 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 538/1058] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:17:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:17:13.297664 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b880fe4-1cdb-4f87-b660-bc2b147cca53 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:17:13.299639 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-3b880fe4-1cdb-4f87-b660-bc2b147cca53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:13.478075 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:13.490775 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b880fe4-1cdb-4f87-b660-bc2b147cca53 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:17:13.629685 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3212838-8095-4705-8862-ea86bdce3750 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:17:13.630229 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3212838-8095-4705-8862-ea86bdce3750 service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:13.630464 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3212838-8095-4705-8862-ea86bdce3750 service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:13.630732 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3212838-8095-4705-8862-ea86bdce3750 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:13.632318 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-b3212838-8095-4705-8862-ea86bdce3750 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:13.730352 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3212838-8095-4705-8862-ea86bdce3750 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:17:13.732652 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3212838-8095-4705-8862-ea86bdce3750 service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:17:13.747623 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:13.757763 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 521/1059] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:17:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 461 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:17:15.952937 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c07875-ac61-4b65-af02-087390d1a136 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:15.953639 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c07875-ac61-4b65-af02-087390d1a136 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:15.953639 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c07875-ac61-4b65-af02-087390d1a136 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:15.954837 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 539/1060] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:17:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:17:15.960756 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14516a8e-c2c0-4977-a0ef-deb6f688ff6d None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:15.961253 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14516a8e-c2c0-4977-a0ef-deb6f688ff6d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:15.961354 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14516a8e-c2c0-4977-a0ef-deb6f688ff6d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:16.029174 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-14516a8e-c2c0-4977-a0ef-deb6f688ff6d None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:17:16.030480 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-14516a8e-c2c0-4977-a0ef-deb6f688ff6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:16.138918 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 522/1061] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:17:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:17:16.435655 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c07875-ac61-4b65-af02-087390d1a136 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:17:16.437202 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-24c07875-ac61-4b65-af02-087390d1a136 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:16.615089 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c07875-ac61-4b65-af02-087390d1a136 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:17:16.687414 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:16.744224 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9c0684-e4f7-4bd0-ba7c-1c502fef6d38 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:17:16.744836 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9c0684-e4f7-4bd0-ba7c-1c502fef6d38 service nova] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:16.745151 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9c0684-e4f7-4bd0-ba7c-1c502fef6d38 service nova] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:16.745427 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9c0684-e4f7-4bd0-ba7c-1c502fef6d38 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:16.747023 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-af9c0684-e4f7-4bd0-ba7c-1c502fef6d38 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:16.847253 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9c0684-e4f7-4bd0-ba7c-1c502fef6d38 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:17:16.848914 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9c0684-e4f7-4bd0-ba7c-1c502fef6d38 service nova] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:17:16.867353 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 540/1062] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:17:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 432 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:17:19.171385 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d149ebb-3b22-47ca-941c-7eba19935129 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:19.171976 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d149ebb-3b22-47ca-941c-7eba19935129 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:19.171976 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d149ebb-3b22-47ca-941c-7eba19935129 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:19.173454 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 523/1063] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:17:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:17:19.179992 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772a6969-2e70-4044-be95-967cec490233 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:19.180463 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772a6969-2e70-4044-be95-967cec490233 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:19.180564 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772a6969-2e70-4044-be95-967cec490233 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:19.249051 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-772a6969-2e70-4044-be95-967cec490233 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:17:19.250256 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-772a6969-2e70-4044-be95-967cec490233 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:19.283546 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:19.347605 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 541/1064] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:17:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:17:19.643457 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d149ebb-3b22-47ca-941c-7eba19935129 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:17:19.646050 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-6d149ebb-3b22-47ca-941c-7eba19935129 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:19.722715 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:19.791360 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d149ebb-3b22-47ca-941c-7eba19935129 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:17:19.907997 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733eb4b0-6a0b-4df9-898f-973073108781 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:17:19.908525 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733eb4b0-6a0b-4df9-898f-973073108781 service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:19.908801 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733eb4b0-6a0b-4df9-898f-973073108781 service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:19.909087 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733eb4b0-6a0b-4df9-898f-973073108781 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:19.910769 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-733eb4b0-6a0b-4df9-898f-973073108781 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:19.951285 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:20.004291 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-733eb4b0-6a0b-4df9-898f-973073108781 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:17:20.005916 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-733eb4b0-6a0b-4df9-898f-973073108781 service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:17:20.024165 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 524/1065] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:17:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:17:20.887791 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee5ba10-40c7-4c04-9444-50cd6a1e5b53 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:20.888396 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee5ba10-40c7-4c04-9444-50cd6a1e5b53 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:20.888396 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee5ba10-40c7-4c04-9444-50cd6a1e5b53 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:20.889687 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 542/1066] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:17:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:17:20.895898 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306eb866-7db0-486d-8d77-40dc964aa79b None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:20.896334 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306eb866-7db0-486d-8d77-40dc964aa79b None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:20.896442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306eb866-7db0-486d-8d77-40dc964aa79b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:20.966788 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-306eb866-7db0-486d-8d77-40dc964aa79b None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:17:20.968013 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-306eb866-7db0-486d-8d77-40dc964aa79b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:21.063344 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 525/1067] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:17:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:17:21.372885 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f13981-70d4-402f-b68c-2ef68127e48b None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:21.373202 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f13981-70d4-402f-b68c-2ef68127e48b None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:21.373474 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f13981-70d4-402f-b68c-2ef68127e48b None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:21.374804 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 543/1068] 10.222.0.42 () {54 vars in 962 bytes} [Thu Oct 27 01:17:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:17:21.380646 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c86e07-41f7-4af7-8a9d-04c067957806 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:21.380998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c86e07-41f7-4af7-8a9d-04c067957806 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:21.381455 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c86e07-41f7-4af7-8a9d-04c067957806 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:21.397922 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:21.459604 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-96c86e07-41f7-4af7-8a9d-04c067957806 None None] MFA Rules not processed for user `2880dbb00eec471285d242e8183dd8d0`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:17:21.460802 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-96c86e07-41f7-4af7-8a9d-04c067957806 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:21.535562 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 526/1069] 10.222.0.42 () {58 vars in 1059 bytes} [Thu Oct 27 01:17:21 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:17:21.543288 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f13981-70d4-402f-b68c-2ef68127e48b None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:17:21.544802 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-f0f13981-70d4-402f-b68c-2ef68127e48b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:21.647324 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f13981-70d4-402f-b68c-2ef68127e48b None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:17:21.706362 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:21.722758 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fca0a5c-baa7-4af1-b09f-71d73c5e3cad service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2880dbb00eec471285d242e8183dd8d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:17:21.723351 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fca0a5c-baa7-4af1-b09f-71d73c5e3cad service glance] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:21.723623 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fca0a5c-baa7-4af1-b09f-71d73c5e3cad service glance] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:21.723888 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fca0a5c-baa7-4af1-b09f-71d73c5e3cad service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:21.725498 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-6fca0a5c-baa7-4af1-b09f-71d73c5e3cad service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:21.836686 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fca0a5c-baa7-4af1-b09f-71d73c5e3cad service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:17:21.838147 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fca0a5c-baa7-4af1-b09f-71d73c5e3cad service glance] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:17:21.858516 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 544/1070] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:17:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:17:22.224258 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6025245-1c5a-4401-9cb8-97d94fc23e06 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:22.224789 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6025245-1c5a-4401-9cb8-97d94fc23e06 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:22.224789 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6025245-1c5a-4401-9cb8-97d94fc23e06 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:22.226106 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 527/1071] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:17:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:17:22.231494 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97326bc8-bc57-47a9-b085-cec851ebaea8 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:22.231956 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97326bc8-bc57-47a9-b085-cec851ebaea8 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:22.232068 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97326bc8-bc57-47a9-b085-cec851ebaea8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:22.310512 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-97326bc8-bc57-47a9-b085-cec851ebaea8 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:17:22.311694 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-97326bc8-bc57-47a9-b085-cec851ebaea8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:22.352446 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:22.410542 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 545/1072] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:17:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:17:22.616015 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267b04ea-ca70-40f5-bdba-37f071f7a5df None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:22.616320 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267b04ea-ca70-40f5-bdba-37f071f7a5df None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:22.616595 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267b04ea-ca70-40f5-bdba-37f071f7a5df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:22.684599 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-267b04ea-ca70-40f5-bdba-37f071f7a5df None None] MFA Rules not processed for user `7a3f8b5eb09f4ba29f4bd343bd91f6a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:17:22.685831 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-267b04ea-ca70-40f5-bdba-37f071f7a5df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:22.708905 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97326bc8-bc57-47a9-b085-cec851ebaea8 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:17:22.710101 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-97326bc8-bc57-47a9-b085-cec851ebaea8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:22.742924 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:22.777018 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 528/1073] 10.222.0.42 () {58 vars in 1009 bytes} [Thu Oct 27 01:17:22 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:17:22.821698 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d41b4d-3eb9-470a-9900-01aee6052308 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:22.821983 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d41b4d-3eb9-470a-9900-01aee6052308 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:22.822255 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d41b4d-3eb9-470a-9900-01aee6052308 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:22.823870 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 529/1074] 10.222.0.42 () {54 vars in 943 bytes} [Thu Oct 27 01:17:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 27 01:17:22.829570 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab172351-6b1b-421e-a83a-26bb59871252 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:22.829850 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab172351-6b1b-421e-a83a-26bb59871252 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:22.830117 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab172351-6b1b-421e-a83a-26bb59871252 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:22.837108 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97326bc8-bc57-47a9-b085-cec851ebaea8 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:17:22.894575 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:22.910743 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-ab172351-6b1b-421e-a83a-26bb59871252 None None] MFA Rules not processed for user `77f2cf9e0c3541f9bccfb7e213d88609`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:17:22.911947 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ab172351-6b1b-421e-a83a-26bb59871252 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:22.954513 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d5ba0a-debe-4291-9d32-947ce9e00de4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:17:22.955189 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d5ba0a-debe-4291-9d32-947ce9e00de4 service nova] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:22.955464 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d5ba0a-debe-4291-9d32-947ce9e00de4 service nova] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:22.955730 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d5ba0a-debe-4291-9d32-947ce9e00de4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:22.957958 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-16d5ba0a-debe-4291-9d32-947ce9e00de4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:23.020794 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 530/1075] 10.222.0.42 () {58 vars in 1040 bytes} [Thu Oct 27 01:17:22 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:17:23.030489 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab172351-6b1b-421e-a83a-26bb59871252 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:17:23.039853 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab172351-6b1b-421e-a83a-26bb59871252 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:17:23.048461 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d5ba0a-debe-4291-9d32-947ce9e00de4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:17:23.050969 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d5ba0a-debe-4291-9d32-947ce9e00de4 service nova] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:17:23.066172 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:23.069391 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 546/1076] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:17:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 361 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 27 01:17:23.151097 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1024684a-110a-4bd0-a781-c8aa792f114b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77f2cf9e0c3541f9bccfb7e213d88609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:17:23.151721 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1024684a-110a-4bd0-a781-c8aa792f114b service swift] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:23.152000 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1024684a-110a-4bd0-a781-c8aa792f114b service swift] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:23.152274 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1024684a-110a-4bd0-a781-c8aa792f114b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:23.154067 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-1024684a-110a-4bd0-a781-c8aa792f114b service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:23.251921 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1024684a-110a-4bd0-a781-c8aa792f114b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:17:23.254407 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1024684a-110a-4bd0-a781-c8aa792f114b service swift] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:17:23.263437 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 531/1077] 10.222.0.42 () {60 vars in 1384 bytes} [Thu Oct 27 01:17:23 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 233 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 27 01:17:25.301859 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14de8735-9e20-4992-be8e-f54bf03dc7c9 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:25.302426 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14de8735-9e20-4992-be8e-f54bf03dc7c9 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:25.302426 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14de8735-9e20-4992-be8e-f54bf03dc7c9 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:25.303955 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 547/1078] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:17:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:17:25.309288 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f49595c-61b4-4444-93a4-1605ebf886d7 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:25.310056 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f49595c-61b4-4444-93a4-1605ebf886d7 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:25.310327 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f49595c-61b4-4444-93a4-1605ebf886d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:25.389614 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-8f49595c-61b4-4444-93a4-1605ebf886d7 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:17:25.390995 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-8f49595c-61b4-4444-93a4-1605ebf886d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:25.399319 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:25.501961 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 532/1079] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:17:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:17:25.808628 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14de8735-9e20-4992-be8e-f54bf03dc7c9 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:17:25.812920 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-14de8735-9e20-4992-be8e-f54bf03dc7c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:25.886699 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:25.986420 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14de8735-9e20-4992-be8e-f54bf03dc7c9 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:17:26.115280 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2aba53-c832-4c36-979e-165644e04675 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:17:26.115796 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2aba53-c832-4c36-979e-165644e04675 service nova] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:26.116076 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2aba53-c832-4c36-979e-165644e04675 service nova] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:26.116349 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2aba53-c832-4c36-979e-165644e04675 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:26.118008 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-2e2aba53-c832-4c36-979e-165644e04675 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:26.152092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:26.225658 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e2aba53-c832-4c36-979e-165644e04675 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:17:26.227567 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e2aba53-c832-4c36-979e-165644e04675 service nova] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:17:26.246901 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 548/1080] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:17:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 439 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:17:28.505910 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2587a530-93f7-452f-bd42-4937164e450e None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:28.506516 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2587a530-93f7-452f-bd42-4937164e450e None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:28.506516 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2587a530-93f7-452f-bd42-4937164e450e None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:28.507990 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 533/1081] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:17:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:17:28.513318 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ad2acb-92b9-4f17-8978-0a3086928d33 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:28.513775 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ad2acb-92b9-4f17-8978-0a3086928d33 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:28.513889 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ad2acb-92b9-4f17-8978-0a3086928d33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:28.590220 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-c3ad2acb-92b9-4f17-8978-0a3086928d33 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:17:28.591526 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-c3ad2acb-92b9-4f17-8978-0a3086928d33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:28.714086 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 549/1082] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:17:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:17:29.010637 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2587a530-93f7-452f-bd42-4937164e450e None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:17:29.012420 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-2587a530-93f7-452f-bd42-4937164e450e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:29.167169 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:29.180305 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2587a530-93f7-452f-bd42-4937164e450e None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:17:29.311407 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafc2cfb-b999-4d91-a364-05e93c4e305d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:17:29.314514 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafc2cfb-b999-4d91-a364-05e93c4e305d service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:29.315561 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafc2cfb-b999-4d91-a364-05e93c4e305d service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:29.316467 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafc2cfb-b999-4d91-a364-05e93c4e305d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:29.318156 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-aafc2cfb-b999-4d91-a364-05e93c4e305d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:29.420111 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aafc2cfb-b999-4d91-a364-05e93c4e305d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:17:29.421646 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aafc2cfb-b999-4d91-a364-05e93c4e305d service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:17:29.434721 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:29.447612 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 534/1083] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:17:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 437 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:17:31.709440 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a28f314-4dc8-4a1d-85d3-e8c344f0276f None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:31.710003 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a28f314-4dc8-4a1d-85d3-e8c344f0276f None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:31.710003 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a28f314-4dc8-4a1d-85d3-e8c344f0276f None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:31.711304 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 550/1084] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:17:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 27 01:17:31.717020 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafcdec7-64fa-429b-a885-de189e57edf2 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:31.717479 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafcdec7-64fa-429b-a885-de189e57edf2 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:31.717568 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafcdec7-64fa-429b-a885-de189e57edf2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:31.791375 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-cafcdec7-64fa-429b-a885-de189e57edf2 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:17:31.792665 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-cafcdec7-64fa-429b-a885-de189e57edf2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:31.903621 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 535/1085] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:17:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:17:32.198851 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a28f314-4dc8-4a1d-85d3-e8c344f0276f None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:17:32.200361 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-4a28f314-4dc8-4a1d-85d3-e8c344f0276f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:32.203487 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:32.371305 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a28f314-4dc8-4a1d-85d3-e8c344f0276f None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:17:32.462293 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:32.505750 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1fcc04-236d-4f17-94dc-780edcbcd827 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:17:32.506358 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1fcc04-236d-4f17-94dc-780edcbcd827 service nova] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:32.506634 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1fcc04-236d-4f17-94dc-780edcbcd827 service nova] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:32.506894 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1fcc04-236d-4f17-94dc-780edcbcd827 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:32.508579 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-4e1fcc04-236d-4f17-94dc-780edcbcd827 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:32.609346 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1fcc04-236d-4f17-94dc-780edcbcd827 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:17:32.610950 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1fcc04-236d-4f17-94dc-780edcbcd827 service nova] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:17:32.631377 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 551/1086] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:17:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 433 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:17:41.902125 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=67 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:42.628429 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=61 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:46.431995 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005dcecb-288d-40be-8527-fe9774134a1e None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:46.432577 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005dcecb-288d-40be-8527-fe9774134a1e None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:46.432577 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005dcecb-288d-40be-8527-fe9774134a1e None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:46.433905 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 536/1087] 10.222.0.42 () {54 vars in 918 bytes} [Thu Oct 27 01:17:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:17:46.438919 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3766fe7b-5e39-457e-a00a-4b201eadd2fb None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:46.439320 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3766fe7b-5e39-457e-a00a-4b201eadd2fb None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:46.439457 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3766fe7b-5e39-457e-a00a-4b201eadd2fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:46.514401 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-3766fe7b-5e39-457e-a00a-4b201eadd2fb None None] MFA Rules not processed for user `62051562fc0b4cd08750d2293e71787c`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:17:46.515591 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-3766fe7b-5e39-457e-a00a-4b201eadd2fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:46.621435 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 552/1088] 10.222.0.42 () {58 vars in 1015 bytes} [Thu Oct 27 01:17:46 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:17:46.635927 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938b809d-19b2-40a2-ac24-05448c899b92 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:46.636206 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938b809d-19b2-40a2-ac24-05448c899b92 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:46.636475 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938b809d-19b2-40a2-ac24-05448c899b92 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:46.637878 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 537/1089] 10.222.0.42 () {54 vars in 918 bytes} [Thu Oct 27 01:17:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:17:46.642279 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a614fcb-dfbf-4e0f-a6a6-2ac5693949a0 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:46.642557 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a614fcb-dfbf-4e0f-a6a6-2ac5693949a0 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:46.642897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a614fcb-dfbf-4e0f-a6a6-2ac5693949a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:46.717143 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:46.720240 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-8a614fcb-dfbf-4e0f-a6a6-2ac5693949a0 None None] MFA Rules not processed for user `51033d56f85f4282a49ded6338440677`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:17:46.723032 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-8a614fcb-dfbf-4e0f-a6a6-2ac5693949a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:46.838702 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 553/1090] 10.222.0.42 () {58 vars in 1015 bytes} [Thu Oct 27 01:17:46 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:17:49.401038 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd6b101-8ec8-46e6-9467-5e8fa11b5fde None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:49.401614 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd6b101-8ec8-46e6-9467-5e8fa11b5fde None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:49.401614 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd6b101-8ec8-46e6-9467-5e8fa11b5fde None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:49.402975 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 538/1091] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 27 01:17:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:17:49.408266 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c99fa1-c687-436d-912e-16f4d9f99843 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:49.408787 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c99fa1-c687-436d-912e-16f4d9f99843 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:49.408787 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c99fa1-c687-436d-912e-16f4d9f99843 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:49.474081 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-e4c99fa1-c687-436d-912e-16f4d9f99843 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:17:49.475277 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-e4c99fa1-c687-436d-912e-16f4d9f99843 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:49.529960 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:49.576570 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 554/1092] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 27 01:17:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:17:49.868336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bd6b101-8ec8-46e6-9467-5e8fa11b5fde None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:17:49.869913 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-5bd6b101-8ec8-46e6-9467-5e8fa11b5fde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:50.012914 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bd6b101-8ec8-46e6-9467-5e8fa11b5fde None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:17:50.084130 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:50.123887 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79764fd-ae81-4de2-9833-900ec81e4dc6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:17:50.124486 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79764fd-ae81-4de2-9833-900ec81e4dc6 service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:50.124763 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79764fd-ae81-4de2-9833-900ec81e4dc6 service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:50.125038 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79764fd-ae81-4de2-9833-900ec81e4dc6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:50.126647 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e79764fd-ae81-4de2-9833-900ec81e4dc6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:50.215076 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79764fd-ae81-4de2-9833-900ec81e4dc6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:17:50.216649 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79764fd-ae81-4de2-9833-900ec81e4dc6 service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:17:50.234783 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 539/1093] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:17:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 367 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:17:51.359412 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56087ef1-676b-45de-b16f-384c449df8f2 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:51.359999 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56087ef1-676b-45de-b16f-384c449df8f2 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:51.359999 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56087ef1-676b-45de-b16f-384c449df8f2 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:51.361325 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 555/1094] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 27 01:17:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:17:51.367160 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfa71d0-3c33-4ece-b461-150d9667e7df None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:51.367620 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfa71d0-3c33-4ece-b461-150d9667e7df None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:51.367732 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfa71d0-3c33-4ece-b461-150d9667e7df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:51.442355 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-5bfa71d0-3c33-4ece-b461-150d9667e7df None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:17:51.443554 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-5bfa71d0-3c33-4ece-b461-150d9667e7df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:51.470022 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:51.540058 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 540/1095] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 27 01:17:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:17:51.699601 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56087ef1-676b-45de-b16f-384c449df8f2 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:17:51.701144 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-56087ef1-676b-45de-b16f-384c449df8f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:51.848329 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56087ef1-676b-45de-b16f-384c449df8f2 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:17:51.888530 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:51.969277 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f31a71b-5af6-4f89-8cdf-9fa3ab97878e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:17:51.969800 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f31a71b-5af6-4f89-8cdf-9fa3ab97878e service nova] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:51.970076 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f31a71b-5af6-4f89-8cdf-9fa3ab97878e service nova] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:51.970346 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f31a71b-5af6-4f89-8cdf-9fa3ab97878e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:51.972427 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-9f31a71b-5af6-4f89-8cdf-9fa3ab97878e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:52.060304 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f31a71b-5af6-4f89-8cdf-9fa3ab97878e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:17:52.061905 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f31a71b-5af6-4f89-8cdf-9fa3ab97878e service nova] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:17:52.079162 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 556/1096] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:17:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 380 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:17:53.194305 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b28d31a-b3f8-4392-ae6e-719be5274308 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:53.194834 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b28d31a-b3f8-4392-ae6e-719be5274308 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:53.194834 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b28d31a-b3f8-4392-ae6e-719be5274308 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:53.196222 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 541/1097] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 27 01:17:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:17:53.200968 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cd47c9-8a82-488e-94c6-db6d5bc0fbbe None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:53.201407 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cd47c9-8a82-488e-94c6-db6d5bc0fbbe None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:53.201519 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cd47c9-8a82-488e-94c6-db6d5bc0fbbe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:53.256294 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:53.278473 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-78cd47c9-8a82-488e-94c6-db6d5bc0fbbe None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:17:53.279654 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-78cd47c9-8a82-488e-94c6-db6d5bc0fbbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:53.381270 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 557/1098] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 27 01:17:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:17:53.532513 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b28d31a-b3f8-4392-ae6e-719be5274308 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:17:53.534676 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0b28d31a-b3f8-4392-ae6e-719be5274308 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:53.679680 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b28d31a-b3f8-4392-ae6e-719be5274308 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:17:53.691602 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:53.792163 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f64968e-e525-4520-a1a1-38ee982a2649 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:17:53.792677 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f64968e-e525-4520-a1a1-38ee982a2649 service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:53.792956 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f64968e-e525-4520-a1a1-38ee982a2649 service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:53.793256 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f64968e-e525-4520-a1a1-38ee982a2649 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:53.794837 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0f64968e-e525-4520-a1a1-38ee982a2649 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:53.884727 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f64968e-e525-4520-a1a1-38ee982a2649 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:17:53.886356 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f64968e-e525-4520-a1a1-38ee982a2649 service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:17:53.904617 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 542/1099] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:17:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:17:54.994888 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cf49c4-d202-4532-83fb-11144db9deff None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:54.995471 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cf49c4-d202-4532-83fb-11144db9deff None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:54.995471 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cf49c4-d202-4532-83fb-11144db9deff None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:54.996737 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 558/1100] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 27 01:17:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:17:55.001640 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85db0fba-c163-4515-a5b8-08ff0a39729a None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:55.002082 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85db0fba-c163-4515-a5b8-08ff0a39729a None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:55.002216 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85db0fba-c163-4515-a5b8-08ff0a39729a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:55.023086 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:55.078433 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-85db0fba-c163-4515-a5b8-08ff0a39729a None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:17:55.079647 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-85db0fba-c163-4515-a5b8-08ff0a39729a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:55.176243 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 543/1101] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 27 01:17:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:17:55.324339 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99cf49c4-d202-4532-83fb-11144db9deff None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:17:55.325842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-99cf49c4-d202-4532-83fb-11144db9deff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:55.436332 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:55.473467 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99cf49c4-d202-4532-83fb-11144db9deff None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:17:55.583307 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46fe024f-af47-4737-8d21-3d928fb48473 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:17:55.583832 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46fe024f-af47-4737-8d21-3d928fb48473 service nova] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:55.584104 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46fe024f-af47-4737-8d21-3d928fb48473 service nova] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:55.584370 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46fe024f-af47-4737-8d21-3d928fb48473 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:55.585974 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-46fe024f-af47-4737-8d21-3d928fb48473 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:55.659372 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:55.673052 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46fe024f-af47-4737-8d21-3d928fb48473 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:17:55.674849 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46fe024f-af47-4737-8d21-3d928fb48473 service nova] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:17:55.693490 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 559/1102] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:17:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:17:56.802002 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c163ca-b54e-4105-a0d6-6318207265e4 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:56.802490 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c163ca-b54e-4105-a0d6-6318207265e4 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:56.802607 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c163ca-b54e-4105-a0d6-6318207265e4 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:56.803896 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 544/1103] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 27 01:17:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:17:56.808907 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227ce0e5-2076-40bd-94a8-b5b650ba18d0 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:56.809392 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227ce0e5-2076-40bd-94a8-b5b650ba18d0 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:56.809490 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227ce0e5-2076-40bd-94a8-b5b650ba18d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:56.884009 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-227ce0e5-2076-40bd-94a8-b5b650ba18d0 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:17:56.885249 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-227ce0e5-2076-40bd-94a8-b5b650ba18d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:56.979906 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 560/1104] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 27 01:17:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:17:57.133858 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c163ca-b54e-4105-a0d6-6318207265e4 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:17:57.135371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d0c163ca-b54e-4105-a0d6-6318207265e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:57.217812 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:57.279535 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c163ca-b54e-4105-a0d6-6318207265e4 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:17:57.390634 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2ced78-c9cb-446c-a1db-c4c9927762d3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:17:57.391237 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2ced78-c9cb-446c-a1db-c4c9927762d3 service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:57.391513 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2ced78-c9cb-446c-a1db-c4c9927762d3 service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:57.391777 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2ced78-c9cb-446c-a1db-c4c9927762d3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:57.394063 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-3c2ced78-c9cb-446c-a1db-c4c9927762d3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:57.442093 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:57.482305 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2ced78-c9cb-446c-a1db-c4c9927762d3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:17:57.483897 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2ced78-c9cb-446c-a1db-c4c9927762d3 service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:17:57.501791 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 545/1105] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:17:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:17:58.547067 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a819e0-e669-4c3d-be06-5e185d52f84c None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:58.547652 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a819e0-e669-4c3d-be06-5e185d52f84c None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:58.547652 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a819e0-e669-4c3d-be06-5e185d52f84c None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:58.548954 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 561/1106] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 27 01:17:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:17:58.554280 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931bc00a-4ac4-4f7d-9aaf-adcb2199e175 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:58.554711 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931bc00a-4ac4-4f7d-9aaf-adcb2199e175 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:58.554817 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931bc00a-4ac4-4f7d-9aaf-adcb2199e175 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:58.630303 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-931bc00a-4ac4-4f7d-9aaf-adcb2199e175 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:17:58.631527 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-931bc00a-4ac4-4f7d-9aaf-adcb2199e175 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:58.736536 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 546/1107] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 27 01:17:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:17:58.882441 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a819e0-e669-4c3d-be06-5e185d52f84c None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:17:58.883962 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-43a819e0-e669-4c3d-be06-5e185d52f84c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:58.936967 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:59.035441 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a819e0-e669-4c3d-be06-5e185d52f84c None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:17:59.148569 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3bac95-5eee-4d12-8487-5a760e78bd09 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:17:59.149115 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3bac95-5eee-4d12-8487-5a760e78bd09 service nova] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:17:59.149433 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3bac95-5eee-4d12-8487-5a760e78bd09 service nova] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:17:59.149677 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3bac95-5eee-4d12-8487-5a760e78bd09 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:17:59.151307 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-0a3bac95-5eee-4d12-8487-5a760e78bd09 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:17:59.158218 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:17:59.240671 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a3bac95-5eee-4d12-8487-5a760e78bd09 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:17:59.242300 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a3bac95-5eee-4d12-8487-5a760e78bd09 service nova] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:17:59.260503 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 562/1108] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:17:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:18:00.304910 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc122729-7086-48fb-8b52-6e67cae607fa None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:00.305507 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc122729-7086-48fb-8b52-6e67cae607fa None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:00.305507 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc122729-7086-48fb-8b52-6e67cae607fa None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:00.306826 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 547/1109] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 27 01:18:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:18:00.311589 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7c2ec1-25ef-4669-891d-358730a3bb8e None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:00.311998 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7c2ec1-25ef-4669-891d-358730a3bb8e None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:00.312111 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7c2ec1-25ef-4669-891d-358730a3bb8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:00.386997 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-0d7c2ec1-25ef-4669-891d-358730a3bb8e None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:18:00.388230 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-0d7c2ec1-25ef-4669-891d-358730a3bb8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:00.456429 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:18:00.484833 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 563/1110] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 27 01:18:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:18:00.636921 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc122729-7086-48fb-8b52-6e67cae607fa None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:18:00.638368 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-bc122729-7086-48fb-8b52-6e67cae607fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:00.655546 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:18:00.783105 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc122729-7086-48fb-8b52-6e67cae607fa None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:18:00.871913 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:18:00.899059 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df128ee-4238-4824-86c8-f787a87e9151 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:18:00.899616 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df128ee-4238-4824-86c8-f787a87e9151 service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:00.899891 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df128ee-4238-4824-86c8-f787a87e9151 service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:00.900160 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df128ee-4238-4824-86c8-f787a87e9151 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:00.901771 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-6df128ee-4238-4824-86c8-f787a87e9151 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:00.988950 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6df128ee-4238-4824-86c8-f787a87e9151 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:18:00.990555 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6df128ee-4238-4824-86c8-f787a87e9151 service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:18:01.008701 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 548/1111] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:18:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:18:02.063863 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26386960-92a7-4990-964d-d3382fb8d013 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:02.064432 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26386960-92a7-4990-964d-d3382fb8d013 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:02.064432 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26386960-92a7-4990-964d-d3382fb8d013 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:02.065836 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 564/1112] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 27 01:18:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:18:02.070719 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc87940-5888-432d-8bbe-bc89409a3f61 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:02.071164 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc87940-5888-432d-8bbe-bc89409a3f61 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:02.071261 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc87940-5888-432d-8bbe-bc89409a3f61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:02.147413 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-0dc87940-5888-432d-8bbe-bc89409a3f61 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:18:02.148624 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0dc87940-5888-432d-8bbe-bc89409a3f61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:02.189994 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:18:02.244584 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 549/1113] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 27 01:18:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:18:02.396603 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26386960-92a7-4990-964d-d3382fb8d013 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:18:02.398084 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-26386960-92a7-4990-964d-d3382fb8d013 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:02.540671 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26386960-92a7-4990-964d-d3382fb8d013 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:18:02.593739 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:18:02.652420 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a631cf44-e4e5-40cd-8f22-23c998eef131 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:18:02.652943 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a631cf44-e4e5-40cd-8f22-23c998eef131 service nova] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:02.653254 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a631cf44-e4e5-40cd-8f22-23c998eef131 service nova] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:02.653525 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a631cf44-e4e5-40cd-8f22-23c998eef131 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:02.655104 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-a631cf44-e4e5-40cd-8f22-23c998eef131 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:02.743227 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a631cf44-e4e5-40cd-8f22-23c998eef131 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:18:02.744838 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a631cf44-e4e5-40cd-8f22-23c998eef131 service nova] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:18:02.762875 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 565/1114] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:18:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 367 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:18:03.807739 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e06d9dc-f417-4ad9-ab3a-4948ae696cf1 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:03.808280 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e06d9dc-f417-4ad9-ab3a-4948ae696cf1 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:03.808280 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e06d9dc-f417-4ad9-ab3a-4948ae696cf1 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:03.809660 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 550/1115] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 27 01:18:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 27 01:18:03.814507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3e60a3-2684-4f74-94f9-b608bc904b24 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:03.815555 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3e60a3-2684-4f74-94f9-b608bc904b24 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:03.815555 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3e60a3-2684-4f74-94f9-b608bc904b24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:03.890296 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-bb3e60a3-2684-4f74-94f9-b608bc904b24 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:18:03.891520 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-bb3e60a3-2684-4f74-94f9-b608bc904b24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:03.894244 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:18:03.987887 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 566/1116] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 27 01:18:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:18:04.136782 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e06d9dc-f417-4ad9-ab3a-4948ae696cf1 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:18:04.138279 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0e06d9dc-f417-4ad9-ab3a-4948ae696cf1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:04.282532 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e06d9dc-f417-4ad9-ab3a-4948ae696cf1 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:18:04.309388 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:18:04.395075 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-397ab95e-3fc1-45fa-9519-fa07770bf7f5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:18:04.395602 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397ab95e-3fc1-45fa-9519-fa07770bf7f5 service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:04.395877 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397ab95e-3fc1-45fa-9519-fa07770bf7f5 service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:04.396146 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397ab95e-3fc1-45fa-9519-fa07770bf7f5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:04.397754 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-397ab95e-3fc1-45fa-9519-fa07770bf7f5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:04.484600 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-397ab95e-3fc1-45fa-9519-fa07770bf7f5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:18:04.486213 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-397ab95e-3fc1-45fa-9519-fa07770bf7f5 service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:18:04.504730 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 551/1117] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:18:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:18:06.611376 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee859f51-4618-4d32-a4a0-63156819cf94 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:06.611957 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee859f51-4618-4d32-a4a0-63156819cf94 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:06.611957 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee859f51-4618-4d32-a4a0-63156819cf94 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:06.613436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 567/1118] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 27 01:18:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:18:06.618840 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6fc5ec-4693-4825-a040-b054e0d12230 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:06.619283 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6fc5ec-4693-4825-a040-b054e0d12230 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:06.619465 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6fc5ec-4693-4825-a040-b054e0d12230 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:06.661112 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:18:06.686725 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-6c6fc5ec-4693-4825-a040-b054e0d12230 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:18:06.687922 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-6c6fc5ec-4693-4825-a040-b054e0d12230 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:06.782978 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 552/1119] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 27 01:18:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:18:07.072593 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee859f51-4618-4d32-a4a0-63156819cf94 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:18:07.074229 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-ee859f51-4618-4d32-a4a0-63156819cf94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:07.177936 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee859f51-4618-4d32-a4a0-63156819cf94 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:18:07.210261 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:18:07.251326 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c172b7-50af-41a7-8e9b-cfab0a911f46 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:18:07.251914 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c172b7-50af-41a7-8e9b-cfab0a911f46 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:07.252192 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c172b7-50af-41a7-8e9b-cfab0a911f46 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:07.252480 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c172b7-50af-41a7-8e9b-cfab0a911f46 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:07.254571 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-e3c172b7-50af-41a7-8e9b-cfab0a911f46 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:07.362044 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c172b7-50af-41a7-8e9b-cfab0a911f46 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:18:07.363867 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c172b7-50af-41a7-8e9b-cfab0a911f46 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:18:07.381643 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 568/1120] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:18:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:18:09.690697 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea81720a-65a1-4174-9c67-f75227f0e1d2 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:09.691243 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea81720a-65a1-4174-9c67-f75227f0e1d2 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:09.691243 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea81720a-65a1-4174-9c67-f75227f0e1d2 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:09.692610 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 553/1121] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 27 01:18:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:18:09.698040 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b970531d-2429-409f-9fde-ce4e49e436d8 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:09.698468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b970531d-2429-409f-9fde-ce4e49e436d8 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:09.698573 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b970531d-2429-409f-9fde-ce4e49e436d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:09.764013 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-b970531d-2429-409f-9fde-ce4e49e436d8 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:18:09.765235 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-b970531d-2429-409f-9fde-ce4e49e436d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:09.778255 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:18:09.861559 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 569/1122] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 27 01:18:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:18:10.149852 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea81720a-65a1-4174-9c67-f75227f0e1d2 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:18:10.151374 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ea81720a-65a1-4174-9c67-f75227f0e1d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:10.253202 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea81720a-65a1-4174-9c67-f75227f0e1d2 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:18:10.265756 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:18:10.326367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b26c4aec-d009-4565-b495-6b813d8f384b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:18:10.326967 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26c4aec-d009-4565-b495-6b813d8f384b service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:10.327247 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26c4aec-d009-4565-b495-6b813d8f384b service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:10.327519 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26c4aec-d009-4565-b495-6b813d8f384b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:10.329121 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-b26c4aec-d009-4565-b495-6b813d8f384b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:10.437319 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b26c4aec-d009-4565-b495-6b813d8f384b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:18:10.438773 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b26c4aec-d009-4565-b495-6b813d8f384b service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:18:10.457592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 554/1123] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:18:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:18:12.916872 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36ed57e-0093-42b5-a101-aff30bdae69c None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:12.917463 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36ed57e-0093-42b5-a101-aff30bdae69c None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:12.917463 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36ed57e-0093-42b5-a101-aff30bdae69c None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:12.918768 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 570/1124] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 27 01:18:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:18:12.923935 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c09a92a-0822-4775-8bdb-021fc7efee1a None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:12.924350 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c09a92a-0822-4775-8bdb-021fc7efee1a None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:12.924485 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c09a92a-0822-4775-8bdb-021fc7efee1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:12.973346 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:18:12.991493 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-8c09a92a-0822-4775-8bdb-021fc7efee1a None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:18:12.992696 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-8c09a92a-0822-4775-8bdb-021fc7efee1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:13.089088 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 555/1125] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 27 01:18:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:18:13.374435 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e36ed57e-0093-42b5-a101-aff30bdae69c None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:18:13.375972 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-e36ed57e-0093-42b5-a101-aff30bdae69c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:13.480296 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e36ed57e-0093-42b5-a101-aff30bdae69c None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:18:13.520275 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:18:13.554022 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb83d86-787b-42cc-b1e0-ecc5f8c89901 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:18:13.554621 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb83d86-787b-42cc-b1e0-ecc5f8c89901 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:13.554897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb83d86-787b-42cc-b1e0-ecc5f8c89901 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:13.555165 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb83d86-787b-42cc-b1e0-ecc5f8c89901 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:13.556736 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-ebb83d86-787b-42cc-b1e0-ecc5f8c89901 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:13.663265 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb83d86-787b-42cc-b1e0-ecc5f8c89901 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:18:13.664610 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb83d86-787b-42cc-b1e0-ecc5f8c89901 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:18:13.683307 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 571/1126] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:18:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:18:15.976692 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72052bc9-73bc-4b15-ac79-bbec228efd97 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:15.977287 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72052bc9-73bc-4b15-ac79-bbec228efd97 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:15.977287 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72052bc9-73bc-4b15-ac79-bbec228efd97 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:15.978601 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 556/1127] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 27 01:18:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:18:15.983862 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d49a79-da10-432d-95ac-b50d3e8d156b None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:15.984262 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d49a79-da10-432d-95ac-b50d3e8d156b None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:15.984411 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d49a79-da10-432d-95ac-b50d3e8d156b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:16.051667 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-d2d49a79-da10-432d-95ac-b50d3e8d156b None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:18:16.052848 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-d2d49a79-da10-432d-95ac-b50d3e8d156b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:16.073210 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:18:16.148997 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 572/1128] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 27 01:18:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:18:16.439261 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72052bc9-73bc-4b15-ac79-bbec228efd97 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:18:16.441446 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-72052bc9-73bc-4b15-ac79-bbec228efd97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:16.542751 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72052bc9-73bc-4b15-ac79-bbec228efd97 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:18:16.563327 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:18:16.615929 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b809a9-c29e-4a04-bfac-d9535b0e7777 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:18:16.616768 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b809a9-c29e-4a04-bfac-d9535b0e7777 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:16.617069 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b809a9-c29e-4a04-bfac-d9535b0e7777 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:16.617343 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b809a9-c29e-4a04-bfac-d9535b0e7777 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:16.618921 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-c2b809a9-c29e-4a04-bfac-d9535b0e7777 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:16.727148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b809a9-c29e-4a04-bfac-d9535b0e7777 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:18:16.728501 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b809a9-c29e-4a04-bfac-d9535b0e7777 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:18:16.746750 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 557/1129] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:18:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:18:18.959770 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1364a7f8-78b7-478d-bb47-7a0c80b53f64 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:18.960333 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1364a7f8-78b7-478d-bb47-7a0c80b53f64 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:18.960333 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1364a7f8-78b7-478d-bb47-7a0c80b53f64 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:18.961659 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 573/1130] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 27 01:18:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 27 01:18:18.967141 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfe1045-9a3e-49db-a741-c53ea9803e2d None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:18.967545 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfe1045-9a3e-49db-a741-c53ea9803e2d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:18.967687 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfe1045-9a3e-49db-a741-c53ea9803e2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:19.031764 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:18:19.035479 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-4bfe1045-9a3e-49db-a741-c53ea9803e2d None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:18:19.037248 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-4bfe1045-9a3e-49db-a741-c53ea9803e2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:19.132346 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 558/1131] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 27 01:18:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:18:19.416031 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1364a7f8-78b7-478d-bb47-7a0c80b53f64 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:18:19.417664 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-1364a7f8-78b7-478d-bb47-7a0c80b53f64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:19.523397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1364a7f8-78b7-478d-bb47-7a0c80b53f64 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:18:19.572292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:18:19.596637 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda30633-02c7-4875-b63d-947c10cedc34 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:18:19.599631 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda30633-02c7-4875-b63d-947c10cedc34 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:19.599905 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda30633-02c7-4875-b63d-947c10cedc34 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:19.600172 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda30633-02c7-4875-b63d-947c10cedc34 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:19.601778 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-dda30633-02c7-4875-b63d-947c10cedc34 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:19.708873 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda30633-02c7-4875-b63d-947c10cedc34 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:18:19.710311 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda30633-02c7-4875-b63d-947c10cedc34 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:18:19.727677 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 574/1132] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:18:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:18:21.937355 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f57acb-e9db-465f-85a3-3229f40b4b50 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:21.937939 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f57acb-e9db-465f-85a3-3229f40b4b50 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:21.938183 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f57acb-e9db-465f-85a3-3229f40b4b50 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:21.939717 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 559/1133] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 27 01:18:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:18:21.945840 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0968a60-f523-4ecf-a6ab-33ad61975f43 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:21.946287 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0968a60-f523-4ecf-a6ab-33ad61975f43 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:21.946449 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0968a60-f523-4ecf-a6ab-33ad61975f43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:22.011694 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-a0968a60-f523-4ecf-a6ab-33ad61975f43 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:18:22.012891 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-a0968a60-f523-4ecf-a6ab-33ad61975f43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:22.041897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:18:22.108649 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 575/1134] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 27 01:18:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:18:22.400024 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11f57acb-e9db-465f-85a3-3229f40b4b50 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:18:22.401577 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-11f57acb-e9db-465f-85a3-3229f40b4b50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:22.504169 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11f57acb-e9db-465f-85a3-3229f40b4b50 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:18:22.531920 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:18:22.577404 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c66822e2-d365-4c74-9795-15e4f32455a4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:18:22.577996 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c66822e2-d365-4c74-9795-15e4f32455a4 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:22.578269 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c66822e2-d365-4c74-9795-15e4f32455a4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:22.578533 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c66822e2-d365-4c74-9795-15e4f32455a4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:22.580086 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-c66822e2-d365-4c74-9795-15e4f32455a4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:22.689485 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c66822e2-d365-4c74-9795-15e4f32455a4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:18:22.690983 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c66822e2-d365-4c74-9795-15e4f32455a4 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:18:22.708415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 560/1135] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:18:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:18:25.022872 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc673f87-67a2-4683-a99e-8750f308f29e None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:25.023411 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc673f87-67a2-4683-a99e-8750f308f29e None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:25.023411 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc673f87-67a2-4683-a99e-8750f308f29e None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:25.024740 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 576/1136] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 27 01:18:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:18:25.029956 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29615040-a099-492c-8e05-c120b213bfc7 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:25.030375 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29615040-a099-492c-8e05-c120b213bfc7 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:25.030497 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29615040-a099-492c-8e05-c120b213bfc7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:25.097381 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-29615040-a099-492c-8e05-c120b213bfc7 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:18:25.098831 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-29615040-a099-492c-8e05-c120b213bfc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:25.106932 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:18:25.196105 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 561/1137] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 27 01:18:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:18:25.480518 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc673f87-67a2-4683-a99e-8750f308f29e None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:18:25.482110 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-dc673f87-67a2-4683-a99e-8750f308f29e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:25.583539 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc673f87-67a2-4683-a99e-8750f308f29e None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:18:25.638726 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:18:25.657460 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c6f766-a5f4-4a2c-9d94-c38cc38af1b2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:18:25.658054 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c6f766-a5f4-4a2c-9d94-c38cc38af1b2 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:25.658324 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c6f766-a5f4-4a2c-9d94-c38cc38af1b2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:25.658583 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c6f766-a5f4-4a2c-9d94-c38cc38af1b2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:25.660331 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-75c6f766-a5f4-4a2c-9d94-c38cc38af1b2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:25.771603 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c6f766-a5f4-4a2c-9d94-c38cc38af1b2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:18:25.773039 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c6f766-a5f4-4a2c-9d94-c38cc38af1b2 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:18:25.791670 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 577/1138] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:18:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:18:27.997466 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db09dda-e37f-47c4-82f9-5b1da7cd8cf5 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:27.998036 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db09dda-e37f-47c4-82f9-5b1da7cd8cf5 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:27.998036 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db09dda-e37f-47c4-82f9-5b1da7cd8cf5 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:27.999293 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 562/1139] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 27 01:18:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:18:28.004534 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b1635c-4d0b-4b51-9fd8-49c1d1f67382 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:28.004981 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b1635c-4d0b-4b51-9fd8-49c1d1f67382 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:28.005108 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b1635c-4d0b-4b51-9fd8-49c1d1f67382 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:28.070389 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-18b1635c-4d0b-4b51-9fd8-49c1d1f67382 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:18:28.071594 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-18b1635c-4d0b-4b51-9fd8-49c1d1f67382 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:28.109351 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:18:28.166302 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 578/1140] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 27 01:18:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:18:28.451675 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db09dda-e37f-47c4-82f9-5b1da7cd8cf5 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:18:28.453236 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-9db09dda-e37f-47c4-82f9-5b1da7cd8cf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:28.555417 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db09dda-e37f-47c4-82f9-5b1da7cd8cf5 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:18:28.592479 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:18:28.627409 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c65eeba-d8db-4451-be41-dba1e24d69a2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:18:28.628528 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c65eeba-d8db-4451-be41-dba1e24d69a2 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:28.628966 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c65eeba-d8db-4451-be41-dba1e24d69a2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:28.629261 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c65eeba-d8db-4451-be41-dba1e24d69a2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:28.630818 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-9c65eeba-d8db-4451-be41-dba1e24d69a2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:28.742992 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c65eeba-d8db-4451-be41-dba1e24d69a2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:18:28.744365 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c65eeba-d8db-4451-be41-dba1e24d69a2 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:18:28.762550 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 563/1141] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:18:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:18:30.944913 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec633b4-7502-4de7-b17a-be3388178095 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:30.945497 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec633b4-7502-4de7-b17a-be3388178095 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:30.945497 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec633b4-7502-4de7-b17a-be3388178095 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:30.946819 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 579/1142] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 27 01:18:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:18:30.952185 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7dab945-be5e-4a20-b607-058d97784218 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:30.952637 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7dab945-be5e-4a20-b607-058d97784218 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:30.952745 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7dab945-be5e-4a20-b607-058d97784218 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:31.017488 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-d7dab945-be5e-4a20-b607-058d97784218 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:18:31.019259 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d7dab945-be5e-4a20-b607-058d97784218 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:31.037296 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:18:31.116817 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 564/1143] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 27 01:18:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:18:31.405951 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec633b4-7502-4de7-b17a-be3388178095 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:18:31.407485 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-9ec633b4-7502-4de7-b17a-be3388178095 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:31.511702 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec633b4-7502-4de7-b17a-be3388178095 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:18:31.578451 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:18:31.585989 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbffddf7-9c19-4e95-b7d9-8956fb0fc0cd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:18:31.586592 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbffddf7-9c19-4e95-b7d9-8956fb0fc0cd service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:31.586868 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbffddf7-9c19-4e95-b7d9-8956fb0fc0cd service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:31.587142 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbffddf7-9c19-4e95-b7d9-8956fb0fc0cd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:31.588729 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-cbffddf7-9c19-4e95-b7d9-8956fb0fc0cd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:31.697688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbffddf7-9c19-4e95-b7d9-8956fb0fc0cd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:18:31.699107 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbffddf7-9c19-4e95-b7d9-8956fb0fc0cd service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:18:31.717086 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 580/1144] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:18:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:18:33.958702 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd70a511-df9b-45b2-b25f-a3ce0addcf0d None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:33.959275 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd70a511-df9b-45b2-b25f-a3ce0addcf0d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:33.959275 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd70a511-df9b-45b2-b25f-a3ce0addcf0d None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:33.960560 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 565/1145] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 27 01:18:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:18:33.966061 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91808438-faa8-40a0-bddd-b5303ca9b11b None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:33.966488 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91808438-faa8-40a0-bddd-b5303ca9b11b None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:33.966592 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91808438-faa8-40a0-bddd-b5303ca9b11b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:34.032539 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-91808438-faa8-40a0-bddd-b5303ca9b11b None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:18:34.033765 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-91808438-faa8-40a0-bddd-b5303ca9b11b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:34.083089 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:18:34.131801 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 581/1146] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 27 01:18:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:18:34.419448 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd70a511-df9b-45b2-b25f-a3ce0addcf0d None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:18:34.420974 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-cd70a511-df9b-45b2-b25f-a3ce0addcf0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:34.522985 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd70a511-df9b-45b2-b25f-a3ce0addcf0d None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:18:34.571023 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:18:34.597025 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3a3abf-6ac8-4456-be06-a8af8e7caa27 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:18:34.598015 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3a3abf-6ac8-4456-be06-a8af8e7caa27 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:34.598289 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3a3abf-6ac8-4456-be06-a8af8e7caa27 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:34.598552 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3a3abf-6ac8-4456-be06-a8af8e7caa27 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:34.600101 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-eb3a3abf-6ac8-4456-be06-a8af8e7caa27 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:34.708826 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3a3abf-6ac8-4456-be06-a8af8e7caa27 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:18:34.710280 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3a3abf-6ac8-4456-be06-a8af8e7caa27 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:18:34.728434 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 566/1147] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:18:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:18:44.130347 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=22 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:18:44.726284 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=66 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:18:54.736770 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65074c0f-d2ae-437a-8402-1abd65a96551 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:54.737393 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65074c0f-d2ae-437a-8402-1abd65a96551 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:54.737393 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65074c0f-d2ae-437a-8402-1abd65a96551 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:54.739364 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 582/1148] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:18:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:18:54.744884 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95f1c08-7cbc-429e-b877-701c1130e4ba None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:54.745333 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95f1c08-7cbc-429e-b877-701c1130e4ba None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:54.745457 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95f1c08-7cbc-429e-b877-701c1130e4ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:54.812486 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-c95f1c08-7cbc-429e-b877-701c1130e4ba None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:18:54.813918 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-c95f1c08-7cbc-429e-b877-701c1130e4ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:54.909936 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 567/1149] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:18:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:18:55.208124 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db17324-e9f6-4abf-8b86-05d2b9497164 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:55.208403 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db17324-e9f6-4abf-8b86-05d2b9497164 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:55.208670 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db17324-e9f6-4abf-8b86-05d2b9497164 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:55.210045 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 583/1150] 10.222.0.42 () {54 vars in 962 bytes} [Thu Oct 27 01:18:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:18:55.214989 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977f7a36-c160-4c69-8b84-70674440b2b3 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:55.215342 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977f7a36-c160-4c69-8b84-70674440b2b3 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:55.215621 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977f7a36-c160-4c69-8b84-70674440b2b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:55.290643 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-977f7a36-c160-4c69-8b84-70674440b2b3 None None] MFA Rules not processed for user `2880dbb00eec471285d242e8183dd8d0`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:18:55.291837 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-977f7a36-c160-4c69-8b84-70674440b2b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:55.312600 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:18:55.366676 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 568/1151] 10.222.0.42 () {58 vars in 1059 bytes} [Thu Oct 27 01:18:55 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:18:55.374654 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db17324-e9f6-4abf-8b86-05d2b9497164 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:18:55.376170 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-4db17324-e9f6-4abf-8b86-05d2b9497164 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:55.479342 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db17324-e9f6-4abf-8b86-05d2b9497164 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:18:55.551600 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ba7476-fb34-4063-a09a-f55bebaee16b service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2880dbb00eec471285d242e8183dd8d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:18:55.552188 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ba7476-fb34-4063-a09a-f55bebaee16b service glance] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:55.552731 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ba7476-fb34-4063-a09a-f55bebaee16b service glance] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:55.552998 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ba7476-fb34-4063-a09a-f55bebaee16b service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:55.554608 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-96ba7476-fb34-4063-a09a-f55bebaee16b service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:55.610357 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:18:55.664534 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ba7476-fb34-4063-a09a-f55bebaee16b service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:18:55.666070 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ba7476-fb34-4063-a09a-f55bebaee16b service glance] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:18:55.683889 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 584/1152] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:18:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:18:57.937631 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7a5a01-8e87-4e49-9fae-60e219d173aa None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:57.938190 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7a5a01-8e87-4e49-9fae-60e219d173aa None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:57.938190 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7a5a01-8e87-4e49-9fae-60e219d173aa None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:57.939514 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 569/1153] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:18:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 27 01:18:57.945047 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d6dab8-1c30-41b2-8233-df042d5f5120 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:57.945489 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d6dab8-1c30-41b2-8233-df042d5f5120 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:57.945617 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d6dab8-1c30-41b2-8233-df042d5f5120 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:58.010629 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-53d6dab8-1c30-41b2-8233-df042d5f5120 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:18:58.011823 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-53d6dab8-1c30-41b2-8233-df042d5f5120 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:58.106540 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 585/1154] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:18:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:18:58.383927 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7a5a01-8e87-4e49-9fae-60e219d173aa None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:18:58.385425 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-1b7a5a01-8e87-4e49-9fae-60e219d173aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:18:58.510241 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7a5a01-8e87-4e49-9fae-60e219d173aa None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:18:58.562646 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:18:58.607772 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d883d9-0a0f-4546-bf4f-f2d0fe86cf62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:18:58.608371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d883d9-0a0f-4546-bf4f-f2d0fe86cf62 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:18:58.608646 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d883d9-0a0f-4546-bf4f-f2d0fe86cf62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:18:58.608910 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d883d9-0a0f-4546-bf4f-f2d0fe86cf62 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:18:58.609572 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1d883d9-0a0f-4546-bf4f-f2d0fe86cf62 admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:18:58.610677 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: WARNING py.warnings [None req-f1d883d9-0a0f-4546-bf4f-f2d0fe86cf62 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 27 01:18:58.610677 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: warnings.warn(msg) Oct 27 01:18:58.610677 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: Oct 27 01:18:58.611813 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1d883d9-0a0f-4546-bf4f-f2d0fe86cf62 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:18:58.630675 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 570/1155] 10.222.0.42 () {56 vars in 1108 bytes} [Thu Oct 27 01:18:58 2022] GET /identity/v3/projects => generated 2553 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:19:00.818091 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901d2b27-fd5e-4cb7-9402-9d92ad633a3a None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:19:00.818596 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901d2b27-fd5e-4cb7-9402-9d92ad633a3a None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:19:00.818722 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901d2b27-fd5e-4cb7-9402-9d92ad633a3a None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:19:00.819949 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 586/1156] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:19:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:19:00.825198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d390f37-5ba5-4616-9db8-e9861261bccc None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:19:00.825634 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d390f37-5ba5-4616-9db8-e9861261bccc None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:19:00.825792 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d390f37-5ba5-4616-9db8-e9861261bccc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:19:00.891251 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-4d390f37-5ba5-4616-9db8-e9861261bccc None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:19:00.892455 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-4d390f37-5ba5-4616-9db8-e9861261bccc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:19:00.989845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 571/1157] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:19:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:19:01.280489 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901d2b27-fd5e-4cb7-9402-9d92ad633a3a None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:19:01.282079 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-901d2b27-fd5e-4cb7-9402-9d92ad633a3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:19:01.292426 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:19:01.428728 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901d2b27-fd5e-4cb7-9402-9d92ad633a3a None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:19:01.506379 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:19:01.538938 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fc7770-8f6e-4d53-82e7-19917b9f9d0d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:19:01.539465 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fc7770-8f6e-4d53-82e7-19917b9f9d0d service nova] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:19:01.539742 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fc7770-8f6e-4d53-82e7-19917b9f9d0d service nova] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:19:01.540013 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fc7770-8f6e-4d53-82e7-19917b9f9d0d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:19:01.541627 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-d8fc7770-8f6e-4d53-82e7-19917b9f9d0d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:19:01.631346 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8fc7770-8f6e-4d53-82e7-19917b9f9d0d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:19:01.632937 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8fc7770-8f6e-4d53-82e7-19917b9f9d0d service nova] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:19:01.651185 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 587/1158] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:19:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:19:03.981346 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6622ef73-bf7d-4563-9f79-024a2f52543e None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:19:03.981907 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6622ef73-bf7d-4563-9f79-024a2f52543e None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:19:03.981907 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6622ef73-bf7d-4563-9f79-024a2f52543e None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:19:03.983232 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 572/1159] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:19:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:19:03.989179 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b8a47f-33bd-4d7d-bbe9-4019a43dfc62 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:19:03.989619 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b8a47f-33bd-4d7d-bbe9-4019a43dfc62 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:19:03.989731 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b8a47f-33bd-4d7d-bbe9-4019a43dfc62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:19:04.054255 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-d4b8a47f-33bd-4d7d-bbe9-4019a43dfc62 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:19:04.055482 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-d4b8a47f-33bd-4d7d-bbe9-4019a43dfc62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:19:04.101100 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:19:04.150858 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 588/1160] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:19:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:19:04.435767 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6622ef73-bf7d-4563-9f79-024a2f52543e None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:19:04.437321 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-6622ef73-bf7d-4563-9f79-024a2f52543e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:19:04.460994 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:19:04.539611 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6622ef73-bf7d-4563-9f79-024a2f52543e None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:19:04.610210 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6286383a-c30e-4ea5-ae52-e69b6fb353ec service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:19:04.610807 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6286383a-c30e-4ea5-ae52-e69b6fb353ec service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:19:04.611084 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6286383a-c30e-4ea5-ae52-e69b6fb353ec service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:19:04.611353 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6286383a-c30e-4ea5-ae52-e69b6fb353ec service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:19:04.612947 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-6286383a-c30e-4ea5-ae52-e69b6fb353ec service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:19:04.686639 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:19:04.721364 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6286383a-c30e-4ea5-ae52-e69b6fb353ec service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:19:04.722807 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6286383a-c30e-4ea5-ae52-e69b6fb353ec service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:19:04.740632 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 573/1161] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:19:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:19:06.951307 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149db566-d87e-4563-87d0-2d57c6665fbb None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:19:06.951831 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149db566-d87e-4563-87d0-2d57c6665fbb None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:19:06.951831 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149db566-d87e-4563-87d0-2d57c6665fbb None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:19:06.953203 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 589/1162] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:19:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:19:06.959012 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e82aef-a7c8-4c40-ad6f-fe952b06464c None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:19:06.959465 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e82aef-a7c8-4c40-ad6f-fe952b06464c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:19:06.959576 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e82aef-a7c8-4c40-ad6f-fe952b06464c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:19:07.024612 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-83e82aef-a7c8-4c40-ad6f-fe952b06464c None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:19:07.025850 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-83e82aef-a7c8-4c40-ad6f-fe952b06464c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:19:07.120518 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 574/1163] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:19:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:19:07.409547 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149db566-d87e-4563-87d0-2d57c6665fbb None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:19:07.411613 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-149db566-d87e-4563-87d0-2d57c6665fbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:19:07.515220 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149db566-d87e-4563-87d0-2d57c6665fbb None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:19:07.587583 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:19:07.589010 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8842bbca-75e2-4e16-ad1a-d9fad013a07c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:19:07.589691 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8842bbca-75e2-4e16-ad1a-d9fad013a07c service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:19:07.590157 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8842bbca-75e2-4e16-ad1a-d9fad013a07c service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:19:07.590597 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8842bbca-75e2-4e16-ad1a-d9fad013a07c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:19:07.592584 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-8842bbca-75e2-4e16-ad1a-d9fad013a07c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:19:07.702701 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8842bbca-75e2-4e16-ad1a-d9fad013a07c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:19:07.704131 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8842bbca-75e2-4e16-ad1a-d9fad013a07c service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:19:07.722256 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 590/1164] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:19:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:19:10.061032 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6217c74-2f94-4ee7-a26b-d87cafef47b9 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:19:10.061592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6217c74-2f94-4ee7-a26b-d87cafef47b9 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:19:10.061592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6217c74-2f94-4ee7-a26b-d87cafef47b9 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:19:10.062923 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 575/1165] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:19:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:19:10.068815 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859c4c29-6c34-4e8a-a0ca-3f5d993cf29a None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:19:10.069253 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859c4c29-6c34-4e8a-a0ca-3f5d993cf29a None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:19:10.069356 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859c4c29-6c34-4e8a-a0ca-3f5d993cf29a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:19:10.133621 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-859c4c29-6c34-4e8a-a0ca-3f5d993cf29a None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:19:10.134803 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-859c4c29-6c34-4e8a-a0ca-3f5d993cf29a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:19:10.187737 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:19:10.231246 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 591/1166] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:19:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:19:10.531736 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6217c74-2f94-4ee7-a26b-d87cafef47b9 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:19:10.533353 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d6217c74-2f94-4ee7-a26b-d87cafef47b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:19:10.570181 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:19:10.637667 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6217c74-2f94-4ee7-a26b-d87cafef47b9 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:19:10.711294 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68416611-1cb5-4ccf-88d3-ba8553df2d1b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:19:10.711889 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68416611-1cb5-4ccf-88d3-ba8553df2d1b service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:19:10.712161 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68416611-1cb5-4ccf-88d3-ba8553df2d1b service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:19:10.712422 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68416611-1cb5-4ccf-88d3-ba8553df2d1b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:19:10.714018 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-68416611-1cb5-4ccf-88d3-ba8553df2d1b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:19:10.792021 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:19:10.825911 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68416611-1cb5-4ccf-88d3-ba8553df2d1b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:19:10.827340 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68416611-1cb5-4ccf-88d3-ba8553df2d1b service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:19:10.845015 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 576/1167] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:19:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:19:13.250344 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35db29ae-67a4-4169-9c01-adad8027edb3 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:19:13.250932 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35db29ae-67a4-4169-9c01-adad8027edb3 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:19:13.250932 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35db29ae-67a4-4169-9c01-adad8027edb3 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:19:13.252230 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 592/1168] 10.222.0.42 () {54 vars in 927 bytes} [Thu Oct 27 01:19:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:19:13.258311 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba2f021-8cf2-4414-b89a-d3418d986957 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:19:13.258790 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba2f021-8cf2-4414-b89a-d3418d986957 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:19:13.258909 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba2f021-8cf2-4414-b89a-d3418d986957 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:19:13.324554 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-cba2f021-8cf2-4414-b89a-d3418d986957 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:19:13.325783 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-cba2f021-8cf2-4414-b89a-d3418d986957 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:19:13.420304 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 577/1169] 10.222.0.42 () {58 vars in 1024 bytes} [Thu Oct 27 01:19:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:19:13.714436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35db29ae-67a4-4169-9c01-adad8027edb3 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:19:13.716008 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-35db29ae-67a4-4169-9c01-adad8027edb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:19:13.820646 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35db29ae-67a4-4169-9c01-adad8027edb3 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:19:13.893342 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c093959e-8247-410e-bdaf-e7d572bd3b78 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:19:13.893949 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c093959e-8247-410e-bdaf-e7d572bd3b78 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:19:13.894227 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c093959e-8247-410e-bdaf-e7d572bd3b78 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:19:13.894495 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c093959e-8247-410e-bdaf-e7d572bd3b78 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:19:13.896358 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-c093959e-8247-410e-bdaf-e7d572bd3b78 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:19:13.904842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:19:14.006190 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c093959e-8247-410e-bdaf-e7d572bd3b78 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:19:14.007541 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c093959e-8247-410e-bdaf-e7d572bd3b78 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:19:14.025004 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 593/1170] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:19:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:19:23.418787 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=82 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:19:24.022827 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=50 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:20:27.182472 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c50332-6e1e-4f28-8728-a6b17c4fc5f5 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:20:27.183020 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c50332-6e1e-4f28-8728-a6b17c4fc5f5 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:20:27.183020 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c50332-6e1e-4f28-8728-a6b17c4fc5f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:20:27.258255 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-48c50332-6e1e-4f28-8728-a6b17c4fc5f5 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:20:27.259451 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-48c50332-6e1e-4f28-8728-a6b17c4fc5f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:20:27.354072 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 578/1171] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:20:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:20:27.361950 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169dbe89-7423-4847-b5d7-74243daad4ba None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:20:27.362447 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169dbe89-7423-4847-b5d7-74243daad4ba None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:20:27.362545 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169dbe89-7423-4847-b5d7-74243daad4ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:20:27.436974 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-169dbe89-7423-4847-b5d7-74243daad4ba None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:20:27.438136 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-169dbe89-7423-4847-b5d7-74243daad4ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:20:27.533126 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 594/1172] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:20:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:20:27.537729 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c50332-6e1e-4f28-8728-a6b17c4fc5f5 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:20:27.538924 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-48c50332-6e1e-4f28-8728-a6b17c4fc5f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:20:27.550174 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c50332-6e1e-4f28-8728-a6b17c4fc5f5 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:20:27.610828 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:20:27.643617 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d864b220-a196-4c5c-88bf-2821d0792196 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:20:27.644213 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d864b220-a196-4c5c-88bf-2821d0792196 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:20:27.644488 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d864b220-a196-4c5c-88bf-2821d0792196 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:20:27.644755 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d864b220-a196-4c5c-88bf-2821d0792196 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:20:27.645418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d864b220-a196-4c5c-88bf-2821d0792196 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:20:27.645779 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d864b220-a196-4c5c-88bf-2821d0792196 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:20:27.646886 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: WARNING py.warnings [None req-d864b220-a196-4c5c-88bf-2821d0792196 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 27 01:20:27.646886 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: warnings.warn(msg) Oct 27 01:20:27.646886 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: Oct 27 01:20:27.647770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d864b220-a196-4c5c-88bf-2821d0792196 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:20:27.658478 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 579/1173] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:20:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 121 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:20:27.668417 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-169dbe89-7423-4847-b5d7-74243daad4ba None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:20:27.678145 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-169dbe89-7423-4847-b5d7-74243daad4ba None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:20:27.739162 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:20:27.771832 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e599bde-43a1-4753-8c77-f2415830e885 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:20:27.772421 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e599bde-43a1-4753-8c77-f2415830e885 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:20:27.772696 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e599bde-43a1-4753-8c77-f2415830e885 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:20:27.772959 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e599bde-43a1-4753-8c77-f2415830e885 admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:20:27.773609 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e599bde-43a1-4753-8c77-f2415830e885 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:20:27.774613 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: WARNING py.warnings [None req-7e599bde-43a1-4753-8c77-f2415830e885 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 27 01:20:27.774613 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: warnings.warn(msg) Oct 27 01:20:27.774613 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: Oct 27 01:20:27.775800 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e599bde-43a1-4753-8c77-f2415830e885 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:20:27.832571 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 595/1174] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:20:27 2022] POST /identity/v3/projects => generated 377 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:20:27.837268 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d864b220-a196-4c5c-88bf-2821d0792196 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:20:27.838843 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d864b220-a196-4c5c-88bf-2821d0792196 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:20:27.875024 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d864b220-a196-4c5c-88bf-2821d0792196 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:20:27.971848 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5b7ecb-1d0b-4416-b921-c51c74fb4ff4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:20:27.972439 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5b7ecb-1d0b-4416-b921-c51c74fb4ff4 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:20:27.972712 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5b7ecb-1d0b-4416-b921-c51c74fb4ff4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:20:27.972975 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5b7ecb-1d0b-4416-b921-c51c74fb4ff4 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:20:27.973637 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5b7ecb-1d0b-4416-b921-c51c74fb4ff4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:20:27.975381 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: WARNING py.warnings [None req-3b5b7ecb-1d0b-4416-b921-c51c74fb4ff4 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 27 01:20:27.975381 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: warnings.warn(msg) Oct 27 01:20:27.975381 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: Oct 27 01:20:27.976586 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5b7ecb-1d0b-4416-b921-c51c74fb4ff4 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:20:28.017599 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 580/1175] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:20:27 2022] POST /identity/v3/users => generated 289 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:20:28.022204 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e599bde-43a1-4753-8c77-f2415830e885 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:20:28.023703 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-7e599bde-43a1-4753-8c77-f2415830e885 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:20:28.086414 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e599bde-43a1-4753-8c77-f2415830e885 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:20:28.170179 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:20:28.181197 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-803994da-5d21-4ea8-bb2c-873c2839ecf3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:20:28.181796 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803994da-5d21-4ea8-bb2c-873c2839ecf3 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:20:28.182072 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803994da-5d21-4ea8-bb2c-873c2839ecf3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:20:28.182340 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803994da-5d21-4ea8-bb2c-873c2839ecf3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:20:28.182973 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-803994da-5d21-4ea8-bb2c-873c2839ecf3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:20:28.184589 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-803994da-5d21-4ea8-bb2c-873c2839ecf3 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:20:28.197101 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 596/1176] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:20:28 2022] GET /identity/v3/roles => generated 1420 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:20:28.201250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5b7ecb-1d0b-4416-b921-c51c74fb4ff4 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:20:28.202757 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-3b5b7ecb-1d0b-4416-b921-c51c74fb4ff4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:20:28.214109 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5b7ecb-1d0b-4416-b921-c51c74fb4ff4 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:20:28.239234 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:20:28.308978 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e8cf1a-b856-4b50-aa38-86d0adaf1f2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:20:28.309542 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e8cf1a-b856-4b50-aa38-86d0adaf1f2c admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:20:28.309821 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e8cf1a-b856-4b50-aa38-86d0adaf1f2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:20:28.310091 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e8cf1a-b856-4b50-aa38-86d0adaf1f2c admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:20:28.310723 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88e8cf1a-b856-4b50-aa38-86d0adaf1f2c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:20:28.312311 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88e8cf1a-b856-4b50-aa38-86d0adaf1f2c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:20:28.324410 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 581/1177] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:20:28 2022] GET /identity/v3/roles => generated 1420 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:20:28.328694 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-803994da-5d21-4ea8-bb2c-873c2839ecf3 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:20:28.330320 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-803994da-5d21-4ea8-bb2c-873c2839ecf3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:20:28.363738 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-803994da-5d21-4ea8-bb2c-873c2839ecf3 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:20:28.456089 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f64c41-c6d4-4b98-aacf-b215ec791f4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:20:28.456737 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f64c41-c6d4-4b98-aacf-b215ec791f4e admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:20:28.457016 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f64c41-c6d4-4b98-aacf-b215ec791f4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:20:28.457318 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f64c41-c6d4-4b98-aacf-b215ec791f4e admin admin] PATH_INFO: `/v3/projects/292c274886794cb793b820b849891c53/users/f27e5cc83faa40469d9910a1c2c049f9/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:20:28.480532 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f64c41-c6d4-4b98-aacf-b215ec791f4e admin admin] RBAC: Authorizing `identity:create_grant(project_id=292c274886794cb793b820b849891c53, user_id=f27e5cc83faa40469d9910a1c2c049f9, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:20:28.483210 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f64c41-c6d4-4b98-aacf-b215ec791f4e admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:20:28.506877 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 597/1178] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:20:28 2022] PUT /identity/v3/projects/292c274886794cb793b820b849891c53/users/f27e5cc83faa40469d9910a1c2c049f9/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:20:28.517893 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e41d75-1e4d-4bae-8f1a-0acaa0ce6588 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:20:28.518178 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e41d75-1e4d-4bae-8f1a-0acaa0ce6588 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:20:28.518452 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e41d75-1e4d-4bae-8f1a-0acaa0ce6588 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:20:28.575930 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-33e41d75-1e4d-4bae-8f1a-0acaa0ce6588 None None] MFA Rules not processed for user `f27e5cc83faa40469d9910a1c2c049f9`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:20:28.577087 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-33e41d75-1e4d-4bae-8f1a-0acaa0ce6588 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:20:28.651845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 582/1179] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:20:28 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:20:28.665942 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0874fc-aae5-4396-b796-75bc01306b37 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:20:28.666225 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0874fc-aae5-4396-b796-75bc01306b37 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:20:28.666496 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0874fc-aae5-4396-b796-75bc01306b37 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:20:28.667840 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 598/1180] 10.222.0.42 () {54 vars in 963 bytes} [Thu Oct 27 01:20:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:20:28.672846 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed1236a-e01c-4659-b8dc-cb5ede065532 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:20:28.673159 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed1236a-e01c-4659-b8dc-cb5ede065532 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:20:28.673436 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed1236a-e01c-4659-b8dc-cb5ede065532 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:20:28.706997 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:20:28.752595 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-aed1236a-e01c-4659-b8dc-cb5ede065532 None None] MFA Rules not processed for user `d35a3d40a1924064859562a72bab0f6d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:20:28.753840 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-aed1236a-e01c-4659-b8dc-cb5ede065532 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:20:28.829408 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 583/1181] 10.222.0.42 () {58 vars in 1060 bytes} [Thu Oct 27 01:20:28 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:20:28.837804 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad0874fc-aae5-4396-b796-75bc01306b37 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:20:28.839316 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-ad0874fc-aae5-4396-b796-75bc01306b37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:20:28.890466 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:20:28.946520 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad0874fc-aae5-4396-b796-75bc01306b37 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:20:29.018422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e19fa9-22fd-415d-9864-a6fc639c2b2b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:20:29.019019 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e19fa9-22fd-415d-9864-a6fc639c2b2b service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:20:29.019320 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e19fa9-22fd-415d-9864-a6fc639c2b2b service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:20:29.019586 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e19fa9-22fd-415d-9864-a6fc639c2b2b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:20:29.021903 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-68e19fa9-22fd-415d-9864-a6fc639c2b2b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:20:29.118927 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e19fa9-22fd-415d-9864-a6fc639c2b2b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:20:29.120267 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e19fa9-22fd-415d-9864-a6fc639c2b2b service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:20:29.123691 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:20:29.138827 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 599/1182] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:20:28 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:20:29.184920 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1fcf90-78df-4926-9fe5-594e45f74312 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:20:29.185246 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1fcf90-78df-4926-9fe5-594e45f74312 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:20:29.185518 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1fcf90-78df-4926-9fe5-594e45f74312 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:20:29.186764 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 584/1183] 10.222.0.42 () {52 vars in 830 bytes} [Thu Oct 27 01:20:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:20:29.190314 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e19fa9-22fd-415d-9864-a6fc639c2b2b service cinder] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:20:29.191551 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-68e19fa9-22fd-415d-9864-a6fc639c2b2b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:20:29.288769 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e19fa9-22fd-415d-9864-a6fc639c2b2b service cinder] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:20:29.379810 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb7cf1e-e889-4722-8204-d98f051c4e5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:20:29.380502 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb7cf1e-e889-4722-8204-d98f051c4e5f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:20:29.380679 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb7cf1e-e889-4722-8204-d98f051c4e5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:20:29.380947 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb7cf1e-e889-4722-8204-d98f051c4e5f admin admin] PATH_INFO: `/v3/users/f27e5cc83faa40469d9910a1c2c049f9` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:20:29.393448 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb7cf1e-e889-4722-8204-d98f051c4e5f admin admin] RBAC: Authorizing `identity:delete_user(user_id=f27e5cc83faa40469d9910a1c2c049f9)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:20:29.394462 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: WARNING py.warnings [None req-2bb7cf1e-e889-4722-8204-d98f051c4e5f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 27 01:20:29.394462 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: warnings.warn(msg) Oct 27 01:20:29.394462 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: Oct 27 01:20:29.396214 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb7cf1e-e889-4722-8204-d98f051c4e5f admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:20:29.406149 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:20:29.462904 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-2bb7cf1e-e889-4722-8204-d98f051c4e5f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f27e5cc83faa40469d9910a1c2c049f9'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:20:29.471866 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-2bb7cf1e-e889-4722-8204-d98f051c4e5f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f27e5cc83faa40469d9910a1c2c049f9'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:20:29.495392 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-2bb7cf1e-e889-4722-8204-d98f051c4e5f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f27e5cc83faa40469d9910a1c2c049f9'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:20:29.495927 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-2bb7cf1e-e889-4722-8204-d98f051c4e5f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f27e5cc83faa40469d9910a1c2c049f9'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:20:29.496345 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-2bb7cf1e-e889-4722-8204-d98f051c4e5f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f27e5cc83faa40469d9910a1c2c049f9'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:20:29.515261 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 600/1184] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:20:29 2022] DELETE /identity/v3/users/f27e5cc83faa40469d9910a1c2c049f9 => generated 0 bytes in 325 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:20:29.519759 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e2c73a-df83-476b-81f2-87b56e9d8362 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:20:29.520232 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e2c73a-df83-476b-81f2-87b56e9d8362 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:20:29.520335 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e2c73a-df83-476b-81f2-87b56e9d8362 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:20:29.577524 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-b4e2c73a-df83-476b-81f2-87b56e9d8362 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:20:29.578722 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-b4e2c73a-df83-476b-81f2-87b56e9d8362 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:20:29.655171 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:20:29.667729 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 585/1185] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:20:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:20:29.677664 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb7cf1e-e889-4722-8204-d98f051c4e5f admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:20:29.678883 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-2bb7cf1e-e889-4722-8204-d98f051c4e5f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:20:29.777289 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb7cf1e-e889-4722-8204-d98f051c4e5f admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:20:29.835120 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:20:29.856307 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d65e22b-66b6-453a-9268-3071616f0878 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:20:29.856913 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d65e22b-66b6-453a-9268-3071616f0878 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:20:29.857224 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d65e22b-66b6-453a-9268-3071616f0878 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:20:29.857487 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d65e22b-66b6-453a-9268-3071616f0878 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:20:29.859071 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-8d65e22b-66b6-453a-9268-3071616f0878 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:20:29.969511 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d65e22b-66b6-453a-9268-3071616f0878 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:20:29.970957 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d65e22b-66b6-453a-9268-3071616f0878 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:20:29.989873 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 601/1186] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:20:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:20:30.345084 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e2c73a-df83-476b-81f2-87b56e9d8362 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:20:30.346375 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-b4e2c73a-df83-476b-81f2-87b56e9d8362 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:20:30.359428 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e2c73a-df83-476b-81f2-87b56e9d8362 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:20:30.459125 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955946ca-7b34-4e9b-b60a-633fb95a7042 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:20:30.459784 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955946ca-7b34-4e9b-b60a-633fb95a7042 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:20:30.460059 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955946ca-7b34-4e9b-b60a-633fb95a7042 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:20:30.460325 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955946ca-7b34-4e9b-b60a-633fb95a7042 admin admin] PATH_INFO: `/v3/projects/292c274886794cb793b820b849891c53` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:20:30.470111 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-955946ca-7b34-4e9b-b60a-633fb95a7042 admin admin] RBAC: Authorizing `identity:delete_project(project_id=292c274886794cb793b820b849891c53)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:20:30.471133 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: WARNING py.warnings [None req-955946ca-7b34-4e9b-b60a-633fb95a7042 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 27 01:20:30.471133 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: warnings.warn(msg) Oct 27 01:20:30.471133 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: Oct 27 01:20:30.472324 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-955946ca-7b34-4e9b-b60a-633fb95a7042 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:20:30.530588 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-955946ca-7b34-4e9b-b60a-633fb95a7042 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '292c274886794cb793b820b849891c53', 'request_id': 'req-955946ca-7b34-4e9b-b60a-633fb95a7042'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:20:30.539995 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-955946ca-7b34-4e9b-b60a-633fb95a7042 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '292c274886794cb793b820b849891c53', 'request_id': 'req-955946ca-7b34-4e9b-b60a-633fb95a7042'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:20:30.546057 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-955946ca-7b34-4e9b-b60a-633fb95a7042 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '292c274886794cb793b820b849891c53', 'request_id': 'req-955946ca-7b34-4e9b-b60a-633fb95a7042'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:20:30.576800 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-955946ca-7b34-4e9b-b60a-633fb95a7042 admin admin] The token cache is being invalidate because project 292c274886794cb793b820b849891c53 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:20:30.577106 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-955946ca-7b34-4e9b-b60a-633fb95a7042 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:20:30.577527 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-955946ca-7b34-4e9b-b60a-633fb95a7042 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:20:30.578909 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 586/1187] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:20:30 2022] DELETE /identity/v3/projects/292c274886794cb793b820b849891c53 => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:20:39.987573 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=62 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:20:40.574807 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:21:54.631848 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026cbf20-334c-45ea-82f6-ea3520a8b06b None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:21:54.632452 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026cbf20-334c-45ea-82f6-ea3520a8b06b None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:21:54.632452 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026cbf20-334c-45ea-82f6-ea3520a8b06b None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:21:54.633754 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 602/1188] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 27 01:21:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:21:54.638576 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25d61ff-2a15-4433-8c14-b67603fd825a None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:21:54.639042 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25d61ff-2a15-4433-8c14-b67603fd825a None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:21:54.639150 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25d61ff-2a15-4433-8c14-b67603fd825a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:21:54.719601 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-c25d61ff-2a15-4433-8c14-b67603fd825a None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:21:54.720804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-c25d61ff-2a15-4433-8c14-b67603fd825a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:21:54.826864 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 587/1189] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 27 01:21:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:21:54.986386 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026cbf20-334c-45ea-82f6-ea3520a8b06b None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:21:54.988502 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-026cbf20-334c-45ea-82f6-ea3520a8b06b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:21:55.107580 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026cbf20-334c-45ea-82f6-ea3520a8b06b None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:21:55.112963 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e21fea-3653-45d5-a457-73492b7246f4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:21:55.113585 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e21fea-3653-45d5-a457-73492b7246f4 service nova] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:21:55.113864 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e21fea-3653-45d5-a457-73492b7246f4 service nova] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:21:55.114134 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e21fea-3653-45d5-a457-73492b7246f4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:21:55.123381 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e21fea-3653-45d5-a457-73492b7246f4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:21:55.124974 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e21fea-3653-45d5-a457-73492b7246f4 service nova] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:21:55.134510 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 603/1190] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:21:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:21:56.199936 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06d43a6-0df8-45e9-aafa-2a9fc08a3cda None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:21:56.200486 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06d43a6-0df8-45e9-aafa-2a9fc08a3cda None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:21:56.200486 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06d43a6-0df8-45e9-aafa-2a9fc08a3cda None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:21:56.201859 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 588/1191] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 27 01:21:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:21:56.206685 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112a31f5-51e3-4ef2-b105-3e942c440a8d None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:21:56.207102 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112a31f5-51e3-4ef2-b105-3e942c440a8d None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:21:56.207225 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112a31f5-51e3-4ef2-b105-3e942c440a8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:21:56.227508 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-112a31f5-51e3-4ef2-b105-3e942c440a8d None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:21:56.228675 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-112a31f5-51e3-4ef2-b105-3e942c440a8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:21:56.236323 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 604/1192] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 27 01:21:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:21:56.399183 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06d43a6-0df8-45e9-aafa-2a9fc08a3cda None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:21:56.409173 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc3800a-0f91-4d75-98a0-55aa109727f3 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:21:56.409458 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc3800a-0f91-4d75-98a0-55aa109727f3 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:21:56.409725 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc3800a-0f91-4d75-98a0-55aa109727f3 None None] PATH_INFO: `/v3` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:21:56.410972 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 605/1193] 10.222.0.13 () {54 vars in 925 bytes} [Thu Oct 27 01:21:56 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:21:56.414623 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06d43a6-0df8-45e9-aafa-2a9fc08a3cda None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:21:56.416506 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca977d3-b2f8-4173-b9e6-756e57452964 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:21:56.416784 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca977d3-b2f8-4173-b9e6-756e57452964 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:21:56.417048 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca977d3-b2f8-4173-b9e6-756e57452964 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:21:56.420085 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8089e96-b403-4c5e-b092-63109f1eb716 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:21:56.420677 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8089e96-b403-4c5e-b092-63109f1eb716 service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:21:56.420951 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8089e96-b403-4c5e-b092-63109f1eb716 service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:21:56.421252 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8089e96-b403-4c5e-b092-63109f1eb716 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:21:56.429671 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8089e96-b403-4c5e-b092-63109f1eb716 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:21:56.431512 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8089e96-b403-4c5e-b092-63109f1eb716 service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:21:56.441426 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 589/1194] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:21:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:21:56.481578 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-1ca977d3-b2f8-4173-b9e6-756e57452964 None None] MFA Rules not processed for user `62051562fc0b4cd08750d2293e71787c`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:21:56.482867 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-1ca977d3-b2f8-4173-b9e6-756e57452964 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:21:56.540821 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:21:56.561099 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 606/1195] 10.222.0.13 () {58 vars in 1015 bytes} [Thu Oct 27 01:21:56 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:22:05.295915 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00360a24-4b91-4a55-a730-90bd98b7acf6 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:22:05.296410 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00360a24-4b91-4a55-a730-90bd98b7acf6 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:22:05.296519 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00360a24-4b91-4a55-a730-90bd98b7acf6 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:22:05.297819 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 590/1196] 10.222.0.42 () {54 vars in 961 bytes} [Thu Oct 27 01:22:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:22:05.302745 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70aa170-ac83-46f1-90c3-46b75d58c650 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:22:05.303184 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70aa170-ac83-46f1-90c3-46b75d58c650 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:22:05.303429 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70aa170-ac83-46f1-90c3-46b75d58c650 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:22:05.323265 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-e70aa170-ac83-46f1-90c3-46b75d58c650 None None] MFA Rules not processed for user `62051562fc0b4cd08750d2293e71787c`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:22:05.324465 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-e70aa170-ac83-46f1-90c3-46b75d58c650 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:22:05.333719 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 607/1197] 10.222.0.42 () {58 vars in 1058 bytes} [Thu Oct 27 01:22:05 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:22:05.341299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00360a24-4b91-4a55-a730-90bd98b7acf6 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:22:05.355535 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00360a24-4b91-4a55-a730-90bd98b7acf6 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:22:05.360377 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e20d4e-e0ba-4e6c-aed3-a27e4cbc8658 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62051562fc0b4cd08750d2293e71787c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:22:05.360965 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e20d4e-e0ba-4e6c-aed3-a27e4cbc8658 service placement] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:22:05.361271 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e20d4e-e0ba-4e6c-aed3-a27e4cbc8658 service placement] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:22:05.361542 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e20d4e-e0ba-4e6c-aed3-a27e4cbc8658 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:22:05.369681 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e20d4e-e0ba-4e6c-aed3-a27e4cbc8658 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:22:05.371236 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e20d4e-e0ba-4e6c-aed3-a27e4cbc8658 service placement] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:22:05.380951 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 591/1198] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:22:05 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:22:07.909011 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0366c2f0-9ae0-4f0b-888f-d6ed87e46ce7 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:22:07.909593 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0366c2f0-9ae0-4f0b-888f-d6ed87e46ce7 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:22:07.909711 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0366c2f0-9ae0-4f0b-888f-d6ed87e46ce7 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:22:07.911039 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 608/1199] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 27 01:22:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:22:07.916249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406ac056-b8b4-4af2-b375-359fd365ac8a None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:22:07.916661 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406ac056-b8b4-4af2-b375-359fd365ac8a None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:22:07.916797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406ac056-b8b4-4af2-b375-359fd365ac8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:22:07.937383 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-406ac056-b8b4-4af2-b375-359fd365ac8a None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:22:07.938581 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-406ac056-b8b4-4af2-b375-359fd365ac8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:22:07.947325 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 592/1200] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 27 01:22:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:22:08.102783 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0366c2f0-9ae0-4f0b-888f-d6ed87e46ce7 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:22:08.117744 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0366c2f0-9ae0-4f0b-888f-d6ed87e46ce7 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:22:08.123133 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa3a9022-ad9f-471e-9f98-3e8c617dcfde service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:22:08.123725 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3a9022-ad9f-471e-9f98-3e8c617dcfde service nova] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:22:08.124003 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3a9022-ad9f-471e-9f98-3e8c617dcfde service nova] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:22:08.124275 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3a9022-ad9f-471e-9f98-3e8c617dcfde service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:22:08.132260 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa3a9022-ad9f-471e-9f98-3e8c617dcfde service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:22:08.133873 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa3a9022-ad9f-471e-9f98-3e8c617dcfde service nova] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:22:08.143178 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 609/1201] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:22:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:22:09.197415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c842c1e4-9c5c-4b72-a342-326c73b41285 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:22:09.197965 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c842c1e4-9c5c-4b72-a342-326c73b41285 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:22:09.197965 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c842c1e4-9c5c-4b72-a342-326c73b41285 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:22:09.199282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 593/1202] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 27 01:22:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:22:09.203979 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5a9553-2ee2-4b57-8cbc-ff39533e1eb8 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:22:09.204430 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5a9553-2ee2-4b57-8cbc-ff39533e1eb8 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:22:09.204539 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5a9553-2ee2-4b57-8cbc-ff39533e1eb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:22:09.226249 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-0a5a9553-2ee2-4b57-8cbc-ff39533e1eb8 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:22:09.227424 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-0a5a9553-2ee2-4b57-8cbc-ff39533e1eb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:22:09.235045 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 610/1203] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 27 01:22:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:22:09.389096 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c842c1e4-9c5c-4b72-a342-326c73b41285 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:22:09.404019 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c842c1e4-9c5c-4b72-a342-326c73b41285 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:22:09.409467 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59515800-924e-4599-a46e-9b6a26919f0a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:22:09.410049 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59515800-924e-4599-a46e-9b6a26919f0a service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:22:09.410326 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59515800-924e-4599-a46e-9b6a26919f0a service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:22:09.410595 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59515800-924e-4599-a46e-9b6a26919f0a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:22:09.418651 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59515800-924e-4599-a46e-9b6a26919f0a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:22:09.420233 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59515800-924e-4599-a46e-9b6a26919f0a service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:22:09.429370 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 594/1204] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:22:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:22:13.321932 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f7fcd5-0145-445a-8fe0-0149538493ea None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:22:13.322515 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f7fcd5-0145-445a-8fe0-0149538493ea None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:22:13.322515 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f7fcd5-0145-445a-8fe0-0149538493ea None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:22:13.323819 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 611/1205] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 27 01:22:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:22:13.328945 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857478bb-7bda-46fa-aa58-adaa6def4556 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:22:13.329373 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857478bb-7bda-46fa-aa58-adaa6def4556 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:22:13.329514 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857478bb-7bda-46fa-aa58-adaa6def4556 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:22:13.348932 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-857478bb-7bda-46fa-aa58-adaa6def4556 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:22:13.350254 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-857478bb-7bda-46fa-aa58-adaa6def4556 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:22:13.357932 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 595/1206] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 27 01:22:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:22:13.652114 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f7fcd5-0145-445a-8fe0-0149538493ea None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:22:13.653864 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-22f7fcd5-0145-445a-8fe0-0149538493ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:22:13.731555 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f7fcd5-0145-445a-8fe0-0149538493ea None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:22:13.735108 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb7dd180-b49f-401a-8480-52d0cc97c9f6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:22:13.735700 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7dd180-b49f-401a-8480-52d0cc97c9f6 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:22:13.735978 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7dd180-b49f-401a-8480-52d0cc97c9f6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:22:13.736248 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7dd180-b49f-401a-8480-52d0cc97c9f6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:22:13.745261 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb7dd180-b49f-401a-8480-52d0cc97c9f6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:22:13.746680 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb7dd180-b49f-401a-8480-52d0cc97c9f6 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:22:13.755967 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 612/1207] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:22:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:22:15.903722 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b353548-ef63-402c-9afd-22b8be3031f0 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:22:15.904207 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b353548-ef63-402c-9afd-22b8be3031f0 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:22:15.904303 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b353548-ef63-402c-9afd-22b8be3031f0 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:22:15.905646 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 596/1208] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 27 01:22:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:22:15.910771 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2cec72-05ac-4f7d-a16f-a922513f400f None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:22:15.911264 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2cec72-05ac-4f7d-a16f-a922513f400f None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:22:15.911359 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2cec72-05ac-4f7d-a16f-a922513f400f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:22:15.931589 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-be2cec72-05ac-4f7d-a16f-a922513f400f None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:22:15.932784 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-be2cec72-05ac-4f7d-a16f-a922513f400f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:22:15.940524 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 613/1209] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 27 01:22:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:22:16.227227 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b353548-ef63-402c-9afd-22b8be3031f0 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:22:16.237931 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b353548-ef63-402c-9afd-22b8be3031f0 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:22:16.242092 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0bffe7-49ab-4edb-8dd4-a5a2079d5b99 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:22:16.242680 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0bffe7-49ab-4edb-8dd4-a5a2079d5b99 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:22:16.242957 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0bffe7-49ab-4edb-8dd4-a5a2079d5b99 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:22:16.243229 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0bffe7-49ab-4edb-8dd4-a5a2079d5b99 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:22:16.251899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac0bffe7-49ab-4edb-8dd4-a5a2079d5b99 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:22:16.253368 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac0bffe7-49ab-4edb-8dd4-a5a2079d5b99 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:22:16.262525 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 597/1210] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:22:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:22:20.139694 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfdf0ea-e3e3-4451-80f7-a26ef44bad19 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:22:20.140210 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfdf0ea-e3e3-4451-80f7-a26ef44bad19 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:22:20.140210 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfdf0ea-e3e3-4451-80f7-a26ef44bad19 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:22:20.141870 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 614/1211] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 27 01:22:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:22:20.147136 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233fa3e4-1511-4e23-8a81-5f83c7859c4f None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:22:20.147552 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233fa3e4-1511-4e23-8a81-5f83c7859c4f None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:22:20.147670 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233fa3e4-1511-4e23-8a81-5f83c7859c4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:22:20.155090 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:22:20.170080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-233fa3e4-1511-4e23-8a81-5f83c7859c4f None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:22:20.171268 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-233fa3e4-1511-4e23-8a81-5f83c7859c4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:22:20.179559 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 598/1212] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 27 01:22:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:22:20.482955 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfdf0ea-e3e3-4451-80f7-a26ef44bad19 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:22:20.484632 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-2dfdf0ea-e3e3-4451-80f7-a26ef44bad19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:22:20.563708 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfdf0ea-e3e3-4451-80f7-a26ef44bad19 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:22:20.567361 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52bf63f5-41aa-468a-832f-e358597b9ee4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:22:20.568031 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bf63f5-41aa-468a-832f-e358597b9ee4 service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:22:20.568311 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bf63f5-41aa-468a-832f-e358597b9ee4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:22:20.568580 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bf63f5-41aa-468a-832f-e358597b9ee4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:22:20.577305 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52bf63f5-41aa-468a-832f-e358597b9ee4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:22:20.578732 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52bf63f5-41aa-468a-832f-e358597b9ee4 service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:22:20.587975 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 615/1213] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:22:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:22:22.740196 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ba53cd-1a5e-4a79-8a22-20951e003fb8 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:22:22.740775 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ba53cd-1a5e-4a79-8a22-20951e003fb8 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:22:22.740775 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ba53cd-1a5e-4a79-8a22-20951e003fb8 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:22:22.742175 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 599/1214] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 27 01:22:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:22:22.747469 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9268a431-1f35-432e-90c3-fd0349029b52 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:22:22.747896 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9268a431-1f35-432e-90c3-fd0349029b52 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:22:22.748042 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9268a431-1f35-432e-90c3-fd0349029b52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:22:22.768371 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-9268a431-1f35-432e-90c3-fd0349029b52 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:22:22.769570 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-9268a431-1f35-432e-90c3-fd0349029b52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:22:22.777360 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 616/1215] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 27 01:22:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:22:23.077934 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ba53cd-1a5e-4a79-8a22-20951e003fb8 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:22:23.079603 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-33ba53cd-1a5e-4a79-8a22-20951e003fb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:22:23.092392 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ba53cd-1a5e-4a79-8a22-20951e003fb8 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:22:23.095859 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d97b6a3-ff2e-4a08-8725-627d2a17cf95 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:22:23.096440 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d97b6a3-ff2e-4a08-8725-627d2a17cf95 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:22:23.096715 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d97b6a3-ff2e-4a08-8725-627d2a17cf95 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:22:23.096980 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d97b6a3-ff2e-4a08-8725-627d2a17cf95 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:22:23.106399 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d97b6a3-ff2e-4a08-8725-627d2a17cf95 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:22:23.107815 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d97b6a3-ff2e-4a08-8725-627d2a17cf95 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:22:23.117619 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 600/1216] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:22:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:22:32.763343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=92 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:22:33.112912 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=8 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:23:47.155273 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2a429a-3184-48c7-9c55-48dce747d05f None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:47.156346 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2a429a-3184-48c7-9c55-48dce747d05f None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:47.156864 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2a429a-3184-48c7-9c55-48dce747d05f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:47.198258 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-7c2a429a-3184-48c7-9c55-48dce747d05f None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:23:47.199921 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-7c2a429a-3184-48c7-9c55-48dce747d05f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:47.234932 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 617/1217] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:23:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:23:47.243660 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3ecabc-d7ba-44df-bbc7-6bafffc0b8e5 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:47.244347 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3ecabc-d7ba-44df-bbc7-6bafffc0b8e5 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:47.244347 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3ecabc-d7ba-44df-bbc7-6bafffc0b8e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:47.298274 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-ef3ecabc-d7ba-44df-bbc7-6bafffc0b8e5 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:23:47.299518 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ef3ecabc-d7ba-44df-bbc7-6bafffc0b8e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:47.306875 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b428957-1f88-4781-a95e-70c949540d03 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:47.307548 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b428957-1f88-4781-a95e-70c949540d03 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:47.308304 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b428957-1f88-4781-a95e-70c949540d03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:47.327088 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 601/1218] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:23:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:23:47.328957 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6496308-4efd-45f6-aefc-e6f75803c418 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:47.329450 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6496308-4efd-45f6-aefc-e6f75803c418 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:47.329902 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6496308-4efd-45f6-aefc-e6f75803c418 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:47.366130 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-8b428957-1f88-4781-a95e-70c949540d03 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:23:47.367754 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-8b428957-1f88-4781-a95e-70c949540d03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:47.374142 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-f6496308-4efd-45f6-aefc-e6f75803c418 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:23:47.375417 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f6496308-4efd-45f6-aefc-e6f75803c418 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:47.403110 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 618/1219] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:23:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:23:47.404347 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b428957-1f88-4781-a95e-70c949540d03 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:47.406451 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 602/1220] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:23:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:23:47.408657 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322aa1dc-fbfe-452a-b7b1-81b417458e30 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:47.415233 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322aa1dc-fbfe-452a-b7b1-81b417458e30 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:47.415872 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322aa1dc-fbfe-452a-b7b1-81b417458e30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:47.430371 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b428957-1f88-4781-a95e-70c949540d03 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:47.447258 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e72b7c-ea6b-456d-b17a-fb0ac6e02650 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:47.448249 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e72b7c-ea6b-456d-b17a-fb0ac6e02650 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:47.448753 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e72b7c-ea6b-456d-b17a-fb0ac6e02650 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:47.449283 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e72b7c-ea6b-456d-b17a-fb0ac6e02650 admin admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:47.450159 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e72b7c-ea6b-456d-b17a-fb0ac6e02650 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:23:47.450676 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e72b7c-ea6b-456d-b17a-fb0ac6e02650 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:47.451911 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: WARNING py.warnings [None req-47e72b7c-ea6b-456d-b17a-fb0ac6e02650 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 27 01:23:47.451911 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: warnings.warn(msg) Oct 27 01:23:47.451911 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: Oct 27 01:23:47.453223 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e72b7c-ea6b-456d-b17a-fb0ac6e02650 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:47.460812 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-322aa1dc-fbfe-452a-b7b1-81b417458e30 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:23:47.467971 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-322aa1dc-fbfe-452a-b7b1-81b417458e30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:47.474673 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 619/1221] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:23:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:23:47.476218 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23f20d9-0825-4844-ae2d-f52e9d0518ee None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:47.476749 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23f20d9-0825-4844-ae2d-f52e9d0518ee None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:47.477270 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23f20d9-0825-4844-ae2d-f52e9d0518ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:47.482875 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 603/1222] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:23:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:23:47.484269 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3dfca3-a0aa-49c4-b4ae-640323bbaa24 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:47.484546 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3dfca3-a0aa-49c4-b4ae-640323bbaa24 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:47.484817 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3dfca3-a0aa-49c4-b4ae-640323bbaa24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:47.507052 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-e23f20d9-0825-4844-ae2d-f52e9d0518ee None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:23:47.508278 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-e23f20d9-0825-4844-ae2d-f52e9d0518ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:47.512166 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-2c3dfca3-a0aa-49c4-b4ae-640323bbaa24 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:23:47.513674 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-2c3dfca3-a0aa-49c4-b4ae-640323bbaa24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:47.528410 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 620/1223] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:23:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:23:47.528410 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd1bb29-191a-44dc-bb02-ba59ea8ff7af None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:47.528410 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd1bb29-191a-44dc-bb02-ba59ea8ff7af None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:47.528410 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd1bb29-191a-44dc-bb02-ba59ea8ff7af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:47.533864 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 604/1224] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:23:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:23:47.535152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26b7cfe-40cb-4609-bc6f-7b829fe5276c None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:47.535423 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26b7cfe-40cb-4609-bc6f-7b829fe5276c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:47.536227 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26b7cfe-40cb-4609-bc6f-7b829fe5276c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:47.557961 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-9fd1bb29-191a-44dc-bb02-ba59ea8ff7af None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:23:47.559171 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-9fd1bb29-191a-44dc-bb02-ba59ea8ff7af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:47.564658 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-c26b7cfe-40cb-4609-bc6f-7b829fe5276c None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:23:47.565869 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-c26b7cfe-40cb-4609-bc6f-7b829fe5276c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:47.568303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 621/1225] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:23:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:23:47.569548 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5146eced-277f-4ed0-b9b9-afc431356e0c None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:47.569832 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5146eced-277f-4ed0-b9b9-afc431356e0c None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:47.570105 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5146eced-277f-4ed0-b9b9-afc431356e0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:47.578768 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 605/1226] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:23:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:23:47.579405 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26b7cfe-40cb-4609-bc6f-7b829fe5276c None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:47.589254 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26b7cfe-40cb-4609-bc6f-7b829fe5276c None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:47.594031 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9d3a65-85c9-4a4e-be3f-9b0f3997e6cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:47.594639 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9d3a65-85c9-4a4e-be3f-9b0f3997e6cd admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:47.594918 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9d3a65-85c9-4a4e-be3f-9b0f3997e6cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:47.595185 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9d3a65-85c9-4a4e-be3f-9b0f3997e6cd admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:47.595817 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9d3a65-85c9-4a4e-be3f-9b0f3997e6cd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:47.596834 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: WARNING py.warnings [None req-fb9d3a65-85c9-4a4e-be3f-9b0f3997e6cd admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 27 01:23:47.596834 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: warnings.warn(msg) Oct 27 01:23:47.596834 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: Oct 27 01:23:47.598697 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9d3a65-85c9-4a4e-be3f-9b0f3997e6cd admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:47.600838 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-5146eced-277f-4ed0-b9b9-afc431356e0c None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:23:47.602571 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-5146eced-277f-4ed0-b9b9-afc431356e0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:47.613107 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 622/1227] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:23:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:23:47.613107 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2188393c-9a92-475a-a501-fe342e7bb34a None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:47.613107 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2188393c-9a92-475a-a501-fe342e7bb34a None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:47.613107 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2188393c-9a92-475a-a501-fe342e7bb34a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:47.632859 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-2188393c-9a92-475a-a501-fe342e7bb34a None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:23:47.634099 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-2188393c-9a92-475a-a501-fe342e7bb34a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:47.647345 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 623/1228] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:23:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:23:47.647345 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d74a695-fdea-425c-9f7a-c52b525f2077 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:47.647345 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d74a695-fdea-425c-9f7a-c52b525f2077 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:47.647345 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d74a695-fdea-425c-9f7a-c52b525f2077 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:47.652756 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 606/1229] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:23:47 2022] POST /identity/v3/projects => generated 387 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:23:47.652756 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85a1c38-8a2c-4e44-b71f-e714c44d0e8a None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:47.652756 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85a1c38-8a2c-4e44-b71f-e714c44d0e8a None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:47.652756 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85a1c38-8a2c-4e44-b71f-e714c44d0e8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:47.668982 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-7d74a695-fdea-425c-9f7a-c52b525f2077 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:23:47.670272 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-7d74a695-fdea-425c-9f7a-c52b525f2077 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:47.670844 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-a85a1c38-8a2c-4e44-b71f-e714c44d0e8a None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:23:47.672018 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-a85a1c38-8a2c-4e44-b71f-e714c44d0e8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:47.731098 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 624/1231] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:23:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:23:47.731320 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 607/1231] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:23:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:23:47.731998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85a1c38-8a2c-4e44-b71f-e714c44d0e8a None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:47.732121 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d74a695-fdea-425c-9f7a-c52b525f2077 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:47.742598 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85a1c38-8a2c-4e44-b71f-e714c44d0e8a None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:47.743693 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d74a695-fdea-425c-9f7a-c52b525f2077 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:47.747390 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd461bc-9b32-4ad4-b2db-dc2387f2ee89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:47.747995 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd461bc-9b32-4ad4-b2db-dc2387f2ee89 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:47.748273 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd461bc-9b32-4ad4-b2db-dc2387f2ee89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:47.748472 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9079046c-496f-48f9-8276-5f3073cd6814 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:47.748637 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd461bc-9b32-4ad4-b2db-dc2387f2ee89 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:47.749083 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9079046c-496f-48f9-8276-5f3073cd6814 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:47.749217 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd461bc-9b32-4ad4-b2db-dc2387f2ee89 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:23:47.749429 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9079046c-496f-48f9-8276-5f3073cd6814 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:47.749536 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd461bc-9b32-4ad4-b2db-dc2387f2ee89 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:47.749699 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9079046c-496f-48f9-8276-5f3073cd6814 admin admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:47.750366 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9079046c-496f-48f9-8276-5f3073cd6814 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:23:47.750665 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9079046c-496f-48f9-8276-5f3073cd6814 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:47.751139 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd461bc-9b32-4ad4-b2db-dc2387f2ee89 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:47.752260 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9079046c-496f-48f9-8276-5f3073cd6814 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:47.763655 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 608/1232] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:23:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:23:47.764843 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eae4343-443a-49af-b528-d85c28511490 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:47.765172 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eae4343-443a-49af-b528-d85c28511490 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:47.765455 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eae4343-443a-49af-b528-d85c28511490 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:47.769341 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 625/1233] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:23:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:23:47.770588 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc22264-91ba-4808-88de-78c4b12ad9d8 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:47.770875 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc22264-91ba-4808-88de-78c4b12ad9d8 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:47.771448 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc22264-91ba-4808-88de-78c4b12ad9d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:47.788928 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-3eae4343-443a-49af-b528-d85c28511490 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:23:47.790233 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-3eae4343-443a-49af-b528-d85c28511490 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:47.791293 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-dbc22264-91ba-4808-88de-78c4b12ad9d8 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:23:47.792466 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-dbc22264-91ba-4808-88de-78c4b12ad9d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:47.798512 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 609/1234] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:23:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:23:47.799393 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eae4343-443a-49af-b528-d85c28511490 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:47.806430 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 626/1235] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:23:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:23:47.807510 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc22264-91ba-4808-88de-78c4b12ad9d8 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:47.815895 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eae4343-443a-49af-b528-d85c28511490 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:47.818478 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc22264-91ba-4808-88de-78c4b12ad9d8 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:47.821147 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8afd8f8-3698-4ff9-a15b-5afbe2742455 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:47.821747 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8afd8f8-3698-4ff9-a15b-5afbe2742455 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:47.822021 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8afd8f8-3698-4ff9-a15b-5afbe2742455 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:47.822289 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8afd8f8-3698-4ff9-a15b-5afbe2742455 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:47.822937 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8afd8f8-3698-4ff9-a15b-5afbe2742455 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:23:47.823071 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009228a6-aae0-4a10-b7a9-256080544df0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:47.823236 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8afd8f8-3698-4ff9-a15b-5afbe2742455 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:47.823666 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009228a6-aae0-4a10-b7a9-256080544df0 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:47.823952 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009228a6-aae0-4a10-b7a9-256080544df0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:47.824217 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009228a6-aae0-4a10-b7a9-256080544df0 admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:47.824818 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8afd8f8-3698-4ff9-a15b-5afbe2742455 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:47.824932 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009228a6-aae0-4a10-b7a9-256080544df0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:47.825875 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: WARNING py.warnings [None req-009228a6-aae0-4a10-b7a9-256080544df0 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 27 01:23:47.825875 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: warnings.warn(msg) Oct 27 01:23:47.825875 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: Oct 27 01:23:47.827084 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009228a6-aae0-4a10-b7a9-256080544df0 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:47.837907 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 610/1236] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:23:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:23:47.839006 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8afd8f8-3698-4ff9-a15b-5afbe2742455 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:47.849436 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8afd8f8-3698-4ff9-a15b-5afbe2742455 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:47.851755 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 627/1237] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:23:47 2022] POST /identity/v3/users => generated 294 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:23:47.852978 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be64f7c7-6fe4-4ecd-813f-b17a3613adc8 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:47.853283 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be64f7c7-6fe4-4ecd-813f-b17a3613adc8 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:47.853555 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be64f7c7-6fe4-4ecd-813f-b17a3613adc8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:47.854073 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3084e574-1270-4983-a954-9bf40bbe42c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:47.854710 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3084e574-1270-4983-a954-9bf40bbe42c9 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:47.854949 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3084e574-1270-4983-a954-9bf40bbe42c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:47.855203 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3084e574-1270-4983-a954-9bf40bbe42c9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:47.855866 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3084e574-1270-4983-a954-9bf40bbe42c9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:23:47.856163 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3084e574-1270-4983-a954-9bf40bbe42c9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:47.857783 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3084e574-1270-4983-a954-9bf40bbe42c9 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:47.872959 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 611/1238] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:23:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:23:47.873809 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3084e574-1270-4983-a954-9bf40bbe42c9 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:47.875449 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-be64f7c7-6fe4-4ecd-813f-b17a3613adc8 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:23:47.876551 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-be64f7c7-6fe4-4ecd-813f-b17a3613adc8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:47.887443 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3084e574-1270-4983-a954-9bf40bbe42c9 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:47.887872 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 628/1239] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:23:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:23:47.888751 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be64f7c7-6fe4-4ecd-813f-b17a3613adc8 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:47.892249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16370ca0-545b-4a75-aab8-c8f0ba22e1d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:47.892844 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16370ca0-545b-4a75-aab8-c8f0ba22e1d4 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:47.893154 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16370ca0-545b-4a75-aab8-c8f0ba22e1d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:47.893447 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16370ca0-545b-4a75-aab8-c8f0ba22e1d4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:47.894076 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16370ca0-545b-4a75-aab8-c8f0ba22e1d4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:47.897044 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16370ca0-545b-4a75-aab8-c8f0ba22e1d4 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:47.899088 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be64f7c7-6fe4-4ecd-813f-b17a3613adc8 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:47.903698 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07c72ce-b1d8-46ea-9559-2ec69fc8d803 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:47.904295 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07c72ce-b1d8-46ea-9559-2ec69fc8d803 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:47.904574 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07c72ce-b1d8-46ea-9559-2ec69fc8d803 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:47.904849 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07c72ce-b1d8-46ea-9559-2ec69fc8d803 admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:47.905506 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b07c72ce-b1d8-46ea-9559-2ec69fc8d803 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:47.907396 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b07c72ce-b1d8-46ea-9559-2ec69fc8d803 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:47.924349 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 612/1240] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:23:47 2022] POST /identity/v3/projects => generated 383 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:23:47.925138 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16370ca0-545b-4a75-aab8-c8f0ba22e1d4 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:47.941026 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 629/1241] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:23:47 2022] POST /identity/v3/projects => generated 377 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:23:47.941026 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07c72ce-b1d8-46ea-9559-2ec69fc8d803 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:47.958344 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:23:47.985598 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:23:47.992821 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16370ca0-545b-4a75-aab8-c8f0ba22e1d4 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:48.002900 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07c72ce-b1d8-46ea-9559-2ec69fc8d803 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:48.007794 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940ce62b-7053-4282-93e7-eea00ae3e213 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:48.008398 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940ce62b-7053-4282-93e7-eea00ae3e213 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:48.008676 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940ce62b-7053-4282-93e7-eea00ae3e213 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:48.008946 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940ce62b-7053-4282-93e7-eea00ae3e213 admin admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:48.009628 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940ce62b-7053-4282-93e7-eea00ae3e213 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:23:48.009932 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940ce62b-7053-4282-93e7-eea00ae3e213 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:48.011543 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940ce62b-7053-4282-93e7-eea00ae3e213 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:48.023803 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 630/1242] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:23:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:23:48.024788 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940ce62b-7053-4282-93e7-eea00ae3e213 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:48.034648 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940ce62b-7053-4282-93e7-eea00ae3e213 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:48.040887 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57644db-cba3-4a35-90ae-92c6756666d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:48.041856 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57644db-cba3-4a35-90ae-92c6756666d0 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:48.041856 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57644db-cba3-4a35-90ae-92c6756666d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:48.042084 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57644db-cba3-4a35-90ae-92c6756666d0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:48.042677 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d57644db-cba3-4a35-90ae-92c6756666d0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:48.044561 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d57644db-cba3-4a35-90ae-92c6756666d0 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:48.058153 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36dfb2cd-5e5c-43c8-839c-dec2f9868c62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:48.058763 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dfb2cd-5e5c-43c8-839c-dec2f9868c62 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:48.059042 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dfb2cd-5e5c-43c8-839c-dec2f9868c62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:48.059313 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dfb2cd-5e5c-43c8-839c-dec2f9868c62 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:48.059963 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36dfb2cd-5e5c-43c8-839c-dec2f9868c62 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:23:48.060268 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36dfb2cd-5e5c-43c8-839c-dec2f9868c62 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:48.062080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36dfb2cd-5e5c-43c8-839c-dec2f9868c62 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:48.071918 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 631/1243] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:23:48 2022] POST /identity/v3/projects => generated 377 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:23:48.073028 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57644db-cba3-4a35-90ae-92c6756666d0 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:48.082086 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 613/1244] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:23:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:23:48.082086 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36dfb2cd-5e5c-43c8-839c-dec2f9868c62 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:48.086376 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36dfb2cd-5e5c-43c8-839c-dec2f9868c62 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:48.142532 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57644db-cba3-4a35-90ae-92c6756666d0 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:48.143207 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-321f65e4-e8b0-4c36-a9ae-10fedfa6d3f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:48.143813 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321f65e4-e8b0-4c36-a9ae-10fedfa6d3f2 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:48.144092 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321f65e4-e8b0-4c36-a9ae-10fedfa6d3f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:48.144363 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321f65e4-e8b0-4c36-a9ae-10fedfa6d3f2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:48.145139 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-321f65e4-e8b0-4c36-a9ae-10fedfa6d3f2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:48.147199 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-321f65e4-e8b0-4c36-a9ae-10fedfa6d3f2 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:48.147430 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3126ee-13d3-4c7a-8e9a-0bb2c63482bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:48.148035 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3126ee-13d3-4c7a-8e9a-0bb2c63482bf admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:48.148317 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3126ee-13d3-4c7a-8e9a-0bb2c63482bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:48.148594 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3126ee-13d3-4c7a-8e9a-0bb2c63482bf admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:48.149279 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a3126ee-13d3-4c7a-8e9a-0bb2c63482bf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:48.150852 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a3126ee-13d3-4c7a-8e9a-0bb2c63482bf admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:48.161230 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 632/1245] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:23:48 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:23:48.161230 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3126ee-13d3-4c7a-8e9a-0bb2c63482bf admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:48.172784 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3126ee-13d3-4c7a-8e9a-0bb2c63482bf admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:48.181988 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 614/1246] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:23:48 2022] POST /identity/v3/projects => generated 393 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:23:48.182148 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2356b343-69ee-4dc2-b2cb-437d1e1b4c92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:48.182287 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-321f65e4-e8b0-4c36-a9ae-10fedfa6d3f2 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:48.182384 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2356b343-69ee-4dc2-b2cb-437d1e1b4c92 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:48.182384 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2356b343-69ee-4dc2-b2cb-437d1e1b4c92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:48.182384 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2356b343-69ee-4dc2-b2cb-437d1e1b4c92 admin admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:48.182384 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2356b343-69ee-4dc2-b2cb-437d1e1b4c92 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:23:48.182384 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2356b343-69ee-4dc2-b2cb-437d1e1b4c92 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:48.182797 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2356b343-69ee-4dc2-b2cb-437d1e1b4c92 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:48.202537 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 633/1247] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:23:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:23:48.202537 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2356b343-69ee-4dc2-b2cb-437d1e1b4c92 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:48.205953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2356b343-69ee-4dc2-b2cb-437d1e1b4c92 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:48.241412 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:23:48.247514 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-321f65e4-e8b0-4c36-a9ae-10fedfa6d3f2 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:48.253625 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a46475-d708-4745-97aa-32177b05d6e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:48.254962 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a46475-d708-4745-97aa-32177b05d6e5 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:48.255161 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a46475-d708-4745-97aa-32177b05d6e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:48.255352 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a46475-d708-4745-97aa-32177b05d6e5 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:48.255970 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a46475-d708-4745-97aa-32177b05d6e5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:48.257968 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a46475-d708-4745-97aa-32177b05d6e5 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:48.263696 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bedf4d36-e17b-4fe3-8c87-79d54c932420 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:48.264456 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedf4d36-e17b-4fe3-8c87-79d54c932420 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:48.264832 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedf4d36-e17b-4fe3-8c87-79d54c932420 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:48.265240 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedf4d36-e17b-4fe3-8c87-79d54c932420 admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:48.266037 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bedf4d36-e17b-4fe3-8c87-79d54c932420 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:48.268558 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bedf4d36-e17b-4fe3-8c87-79d54c932420 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:48.280764 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 615/1248] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:23:48 2022] POST /identity/v3/users => generated 292 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:23:48.281590 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a46475-d708-4745-97aa-32177b05d6e5 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:48.291311 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a46475-d708-4745-97aa-32177b05d6e5 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:48.298339 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 634/1249] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:23:48 2022] POST /identity/v3/users => generated 289 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:23:48.298339 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bedf4d36-e17b-4fe3-8c87-79d54c932420 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:48.299282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10453549-b52c-4ad7-907c-a8a69ef84696 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:48.299906 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10453549-b52c-4ad7-907c-a8a69ef84696 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:48.300183 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10453549-b52c-4ad7-907c-a8a69ef84696 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:48.300449 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10453549-b52c-4ad7-907c-a8a69ef84696 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:48.301104 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10453549-b52c-4ad7-907c-a8a69ef84696 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:48.303292 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10453549-b52c-4ad7-907c-a8a69ef84696 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:48.306922 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bedf4d36-e17b-4fe3-8c87-79d54c932420 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:48.311571 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-454a8ff2-d9be-43f9-be58-258fcdc27ef3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:48.312161 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454a8ff2-d9be-43f9-be58-258fcdc27ef3 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:48.312445 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454a8ff2-d9be-43f9-be58-258fcdc27ef3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:48.312714 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454a8ff2-d9be-43f9-be58-258fcdc27ef3 admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:48.313364 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-454a8ff2-d9be-43f9-be58-258fcdc27ef3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:48.315258 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-454a8ff2-d9be-43f9-be58-258fcdc27ef3 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:48.333889 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 616/1250] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:23:48 2022] POST /identity/v3/projects => generated 365 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:23:48.334561 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10453549-b52c-4ad7-907c-a8a69ef84696 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:48.341360 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 635/1251] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:23:48 2022] POST /identity/v3/users => generated 289 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:23:48.342175 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-454a8ff2-d9be-43f9-be58-258fcdc27ef3 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:48.352166 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-454a8ff2-d9be-43f9-be58-258fcdc27ef3 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:48.356802 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:23:48.402552 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10453549-b52c-4ad7-907c-a8a69ef84696 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:48.407187 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b1862e-1377-4a24-b2db-67ef26662d6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:48.407775 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b1862e-1377-4a24-b2db-67ef26662d6c admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:48.408057 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b1862e-1377-4a24-b2db-67ef26662d6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:48.408333 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b1862e-1377-4a24-b2db-67ef26662d6c admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:48.408967 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b1862e-1377-4a24-b2db-67ef26662d6c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:48.409872 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32fc365b-5024-4bec-8197-2d14dd8711b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:48.410505 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32fc365b-5024-4bec-8197-2d14dd8711b4 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:48.410856 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32fc365b-5024-4bec-8197-2d14dd8711b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:48.410962 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b1862e-1377-4a24-b2db-67ef26662d6c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:48.411137 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32fc365b-5024-4bec-8197-2d14dd8711b4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:48.411769 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32fc365b-5024-4bec-8197-2d14dd8711b4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:48.413408 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32fc365b-5024-4bec-8197-2d14dd8711b4 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:48.424609 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 636/1252] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:23:48 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:23:48.426235 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32fc365b-5024-4bec-8197-2d14dd8711b4 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:48.436977 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32fc365b-5024-4bec-8197-2d14dd8711b4 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:48.439540 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 617/1253] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:23:48 2022] POST /identity/v3/projects => generated 385 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:23:48.440233 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b1862e-1377-4a24-b2db-67ef26662d6c admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:48.444544 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd4939f-c94d-4856-b9e4-3f75a9d417bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:48.445170 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd4939f-c94d-4856-b9e4-3f75a9d417bb admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:48.445444 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd4939f-c94d-4856-b9e4-3f75a9d417bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:48.445712 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd4939f-c94d-4856-b9e4-3f75a9d417bb admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:48.446331 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd4939f-c94d-4856-b9e4-3f75a9d417bb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:48.448314 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd4939f-c94d-4856-b9e4-3f75a9d417bb admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:48.471342 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 637/1254] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:23:48 2022] POST /identity/v3/users => generated 297 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:23:48.472010 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd4939f-c94d-4856-b9e4-3f75a9d417bb admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:48.482546 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd4939f-c94d-4856-b9e4-3f75a9d417bb admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:48.507226 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b1862e-1377-4a24-b2db-67ef26662d6c admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:48.512007 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7255eb1b-d936-438e-99c4-3dfeff376867 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:48.512634 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7255eb1b-d936-438e-99c4-3dfeff376867 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:48.512956 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7255eb1b-d936-438e-99c4-3dfeff376867 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:48.513305 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7255eb1b-d936-438e-99c4-3dfeff376867 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:48.513952 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7255eb1b-d936-438e-99c4-3dfeff376867 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:48.516044 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7255eb1b-d936-438e-99c4-3dfeff376867 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:48.540199 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7adfb217-2b18-461a-84ba-f69cf90657dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:48.540832 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7adfb217-2b18-461a-84ba-f69cf90657dc admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:48.541137 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7adfb217-2b18-461a-84ba-f69cf90657dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:48.541410 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7adfb217-2b18-461a-84ba-f69cf90657dc admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:48.542075 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7adfb217-2b18-461a-84ba-f69cf90657dc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:48.543669 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7adfb217-2b18-461a-84ba-f69cf90657dc admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:48.547251 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 618/1255] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:23:48 2022] POST /identity/v3/projects => generated 381 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:23:48.547251 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7255eb1b-d936-438e-99c4-3dfeff376867 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:48.559721 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 638/1256] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:23:48 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:23:48.559721 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7adfb217-2b18-461a-84ba-f69cf90657dc admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:48.567440 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7adfb217-2b18-461a-84ba-f69cf90657dc admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:48.580428 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:23:48.612215 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7255eb1b-d936-438e-99c4-3dfeff376867 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:48.616996 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-865356c0-7674-4d6e-b633-70b589392fff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:48.617627 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865356c0-7674-4d6e-b633-70b589392fff admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:48.617905 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865356c0-7674-4d6e-b633-70b589392fff admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:48.618174 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865356c0-7674-4d6e-b633-70b589392fff admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:48.618816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-865356c0-7674-4d6e-b633-70b589392fff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:48.620409 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-865356c0-7674-4d6e-b633-70b589392fff admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:48.623492 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033d5099-9e3e-4f15-906d-4b95349c7cdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:48.624094 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033d5099-9e3e-4f15-906d-4b95349c7cdc admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:48.624368 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033d5099-9e3e-4f15-906d-4b95349c7cdc admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:48.624635 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033d5099-9e3e-4f15-906d-4b95349c7cdc admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:48.625280 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033d5099-9e3e-4f15-906d-4b95349c7cdc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:48.627609 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033d5099-9e3e-4f15-906d-4b95349c7cdc admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:48.630260 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 619/1257] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:23:48 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:23:48.631054 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-865356c0-7674-4d6e-b633-70b589392fff admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:48.643145 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-865356c0-7674-4d6e-b633-70b589392fff admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:48.649459 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 639/1258] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:23:48 2022] POST /identity/v3/users => generated 283 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:23:48.649762 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7bab515-0fec-4ecd-bf12-be06709ff52c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:48.650421 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bab515-0fec-4ecd-bf12-be06709ff52c admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:48.650535 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033d5099-9e3e-4f15-906d-4b95349c7cdc admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:48.650702 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bab515-0fec-4ecd-bf12-be06709ff52c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:48.650981 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bab515-0fec-4ecd-bf12-be06709ff52c admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:48.651618 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7bab515-0fec-4ecd-bf12-be06709ff52c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:48.653532 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7bab515-0fec-4ecd-bf12-be06709ff52c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:48.662263 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033d5099-9e3e-4f15-906d-4b95349c7cdc admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:48.667238 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 620/1259] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:23:48 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:23:48.667238 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7bab515-0fec-4ecd-bf12-be06709ff52c admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:48.670020 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ebeb97-074b-4c31-a6b0-ea2ef480ca23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:48.670660 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ebeb97-074b-4c31-a6b0-ea2ef480ca23 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:48.670933 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ebeb97-074b-4c31-a6b0-ea2ef480ca23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:48.671201 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ebeb97-074b-4c31-a6b0-ea2ef480ca23 admin admin] PATH_INFO: `/v3/projects/612f14b893ea46228c14d87e0d843e99/users/26d5719e31114f55abdd4b3f1acacdde/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:48.678116 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7bab515-0fec-4ecd-bf12-be06709ff52c admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:48.683396 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad659ab-3da4-4477-b47c-31e3f30e2e82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:48.683972 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad659ab-3da4-4477-b47c-31e3f30e2e82 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:48.684254 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad659ab-3da4-4477-b47c-31e3f30e2e82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:48.684478 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:23:48.684686 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad659ab-3da4-4477-b47c-31e3f30e2e82 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:48.685411 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad659ab-3da4-4477-b47c-31e3f30e2e82 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:48.687877 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad659ab-3da4-4477-b47c-31e3f30e2e82 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:48.688868 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7ebeb97-074b-4c31-a6b0-ea2ef480ca23 admin admin] RBAC: Authorizing `identity:create_grant(project_id=612f14b893ea46228c14d87e0d843e99, user_id=26d5719e31114f55abdd4b3f1acacdde, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:48.692053 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7ebeb97-074b-4c31-a6b0-ea2ef480ca23 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:48.705268 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 640/1260] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:23:48 2022] PUT /identity/v3/projects/612f14b893ea46228c14d87e0d843e99/users/26d5719e31114f55abdd4b3f1acacdde/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:23:48.706204 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ebeb97-074b-4c31-a6b0-ea2ef480ca23 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:48.710424 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 621/1261] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:23:48 2022] POST /identity/v3/users => generated 293 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:23:48.711106 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad659ab-3da4-4477-b47c-31e3f30e2e82 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:48.720807 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad659ab-3da4-4477-b47c-31e3f30e2e82 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:48.777602 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0f8d19-d8fa-499b-8633-cfc6e5bd74ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:48.778458 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ebeb97-074b-4c31-a6b0-ea2ef480ca23 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:48.778730 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0f8d19-d8fa-499b-8633-cfc6e5bd74ed admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:48.779171 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0f8d19-d8fa-499b-8633-cfc6e5bd74ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:48.779446 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0f8d19-d8fa-499b-8633-cfc6e5bd74ed admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:48.780081 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0f8d19-d8fa-499b-8633-cfc6e5bd74ed admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:48.782142 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0f8d19-d8fa-499b-8633-cfc6e5bd74ed admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:48.783490 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e06718-cb77-47a2-a795-4ce05abcb3bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:48.784096 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e06718-cb77-47a2-a795-4ce05abcb3bb admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:48.784377 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e06718-cb77-47a2-a795-4ce05abcb3bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:48.784648 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e06718-cb77-47a2-a795-4ce05abcb3bb admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:48.785322 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e06718-cb77-47a2-a795-4ce05abcb3bb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:48.787642 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e06718-cb77-47a2-a795-4ce05abcb3bb admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:48.801277 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 641/1262] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:23:48 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:23:48.802519 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e06718-cb77-47a2-a795-4ce05abcb3bb admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:48.807037 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 622/1263] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:23:48 2022] POST /identity/v3/users => generated 291 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:23:48.807860 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0f8d19-d8fa-499b-8633-cfc6e5bd74ed admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:48.817934 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0f8d19-d8fa-499b-8633-cfc6e5bd74ed admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:48.819856 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e06718-cb77-47a2-a795-4ce05abcb3bb admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:48.822617 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-087c3201-badd-46dc-a89a-e93afe773fec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:48.823225 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087c3201-badd-46dc-a89a-e93afe773fec admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:48.823501 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087c3201-badd-46dc-a89a-e93afe773fec admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:48.823769 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087c3201-badd-46dc-a89a-e93afe773fec admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:48.824407 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-087c3201-badd-46dc-a89a-e93afe773fec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:48.824638 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb5f48d-c970-43b7-9a80-0010703b7aa7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:48.825272 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb5f48d-c970-43b7-9a80-0010703b7aa7 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:48.825576 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb5f48d-c970-43b7-9a80-0010703b7aa7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:48.825864 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb5f48d-c970-43b7-9a80-0010703b7aa7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:48.826036 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-087c3201-badd-46dc-a89a-e93afe773fec admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:48.826492 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb5f48d-c970-43b7-9a80-0010703b7aa7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:48.828057 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb5f48d-c970-43b7-9a80-0010703b7aa7 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:48.836817 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 623/1264] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:23:48 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:23:48.837726 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-087c3201-badd-46dc-a89a-e93afe773fec admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:48.845743 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 642/1265] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:23:48 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:23:48.845743 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb5f48d-c970-43b7-9a80-0010703b7aa7 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:48.850713 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-087c3201-badd-46dc-a89a-e93afe773fec admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:48.854789 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb5f48d-c970-43b7-9a80-0010703b7aa7 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:48.855267 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1879c0bb-0d27-4b77-a4d6-9c5956db1c45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:48.855852 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1879c0bb-0d27-4b77-a4d6-9c5956db1c45 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:48.856253 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1879c0bb-0d27-4b77-a4d6-9c5956db1c45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:48.856529 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1879c0bb-0d27-4b77-a4d6-9c5956db1c45 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:48.857190 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1879c0bb-0d27-4b77-a4d6-9c5956db1c45 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:48.858781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1879c0bb-0d27-4b77-a4d6-9c5956db1c45 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:48.859575 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e64fca3-2fd8-43c3-be93-92cbdf308294 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:48.860158 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e64fca3-2fd8-43c3-be93-92cbdf308294 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:48.860436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e64fca3-2fd8-43c3-be93-92cbdf308294 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:48.860710 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e64fca3-2fd8-43c3-be93-92cbdf308294 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:48.861372 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e64fca3-2fd8-43c3-be93-92cbdf308294 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:48.862947 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e64fca3-2fd8-43c3-be93-92cbdf308294 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:48.870098 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 624/1266] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:23:48 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:23:48.870898 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1879c0bb-0d27-4b77-a4d6-9c5956db1c45 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:48.873470 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 643/1267] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:23:48 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:23:48.874531 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23e6429-548e-40f8-9481-2f5ff385fb5b None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:48.874815 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23e6429-548e-40f8-9481-2f5ff385fb5b None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:48.875088 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23e6429-548e-40f8-9481-2f5ff385fb5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:48.883917 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1879c0bb-0d27-4b77-a4d6-9c5956db1c45 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:48.888491 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f334b908-08c4-4d6b-80b9-a2e043e57f87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:48.889152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f334b908-08c4-4d6b-80b9-a2e043e57f87 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:48.889435 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f334b908-08c4-4d6b-80b9-a2e043e57f87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:48.889702 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f334b908-08c4-4d6b-80b9-a2e043e57f87 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:48.890329 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f334b908-08c4-4d6b-80b9-a2e043e57f87 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:48.891916 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f334b908-08c4-4d6b-80b9-a2e043e57f87 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:48.902369 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-e23e6429-548e-40f8-9481-2f5ff385fb5b None None] MFA Rules not processed for user `26d5719e31114f55abdd4b3f1acacdde`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:23:48.902813 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 625/1268] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:23:48 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:23:48.903682 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f334b908-08c4-4d6b-80b9-a2e043e57f87 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:48.903807 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-e23e6429-548e-40f8-9481-2f5ff385fb5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:48.913147 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f334b908-08c4-4d6b-80b9-a2e043e57f87 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:48.917887 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74eaeae6-d0b7-472f-8d4a-d22d16a468ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:48.918468 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74eaeae6-d0b7-472f-8d4a-d22d16a468ba admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:48.918744 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74eaeae6-d0b7-472f-8d4a-d22d16a468ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:48.919013 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74eaeae6-d0b7-472f-8d4a-d22d16a468ba admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:48.919675 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74eaeae6-d0b7-472f-8d4a-d22d16a468ba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:48.921376 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74eaeae6-d0b7-472f-8d4a-d22d16a468ba admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:48.932500 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 626/1269] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:23:48 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:23:48.933132 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74eaeae6-d0b7-472f-8d4a-d22d16a468ba admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:48.943264 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74eaeae6-d0b7-472f-8d4a-d22d16a468ba admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:48.947912 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b132cfda-ed67-47c9-a572-483953205f24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:48.948489 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b132cfda-ed67-47c9-a572-483953205f24 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:48.948760 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b132cfda-ed67-47c9-a572-483953205f24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:48.949028 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b132cfda-ed67-47c9-a572-483953205f24 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:48.949803 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b132cfda-ed67-47c9-a572-483953205f24 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:48.951269 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b132cfda-ed67-47c9-a572-483953205f24 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:48.963238 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 627/1270] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:23:48 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:23:48.964108 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b132cfda-ed67-47c9-a572-483953205f24 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:48.974565 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b132cfda-ed67-47c9-a572-483953205f24 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:48.975950 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 644/1271] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:23:48 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:23:48.976807 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23e6429-548e-40f8-9481-2f5ff385fb5b None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:48.979311 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a068c726-1699-4c3c-8db1-b72968324278 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:48.980009 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a068c726-1699-4c3c-8db1-b72968324278 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:48.980286 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a068c726-1699-4c3c-8db1-b72968324278 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:48.980571 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a068c726-1699-4c3c-8db1-b72968324278 admin admin] PATH_INFO: `/v3/projects/c72a2266373f44749fa0d3bbd3bb68cd/users/68909cf0c5bf4875927b30d9a6ce8cd9/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:48.986672 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23e6429-548e-40f8-9481-2f5ff385fb5b None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:48.991674 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475e7006-993f-476a-ba87-f3ecfddc7f24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:48.992334 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475e7006-993f-476a-ba87-f3ecfddc7f24 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:48.992608 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475e7006-993f-476a-ba87-f3ecfddc7f24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:48.992876 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475e7006-993f-476a-ba87-f3ecfddc7f24 admin admin] PATH_INFO: `/v3/projects/1de72905e70a4b7ab0bc68459daac163/users/931e14007f454ec9be4b44c39b3a9e22/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:48.996339 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a068c726-1699-4c3c-8db1-b72968324278 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c72a2266373f44749fa0d3bbd3bb68cd, user_id=68909cf0c5bf4875927b30d9a6ce8cd9, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:48.999016 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a068c726-1699-4c3c-8db1-b72968324278 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:49.008899 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-475e7006-993f-476a-ba87-f3ecfddc7f24 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1de72905e70a4b7ab0bc68459daac163, user_id=931e14007f454ec9be4b44c39b3a9e22, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:49.011633 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-475e7006-993f-476a-ba87-f3ecfddc7f24 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:49.013085 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 628/1272] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:23:48 2022] PUT /identity/v3/projects/c72a2266373f44749fa0d3bbd3bb68cd/users/68909cf0c5bf4875927b30d9a6ce8cd9/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:23:49.014015 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a068c726-1699-4c3c-8db1-b72968324278 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:49.024991 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 645/1273] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:23:48 2022] PUT /identity/v3/projects/1de72905e70a4b7ab0bc68459daac163/users/931e14007f454ec9be4b44c39b3a9e22/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:23:49.025875 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475e7006-993f-476a-ba87-f3ecfddc7f24 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:49.027197 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:23:49.061341 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:23:49.080887 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a068c726-1699-4c3c-8db1-b72968324278 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:49.096390 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475e7006-993f-476a-ba87-f3ecfddc7f24 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:49.101947 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe35385-f120-4e14-b1c8-42092b7fde10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:49.102778 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe35385-f120-4e14-b1c8-42092b7fde10 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:49.103055 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe35385-f120-4e14-b1c8-42092b7fde10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:49.103515 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe35385-f120-4e14-b1c8-42092b7fde10 admin admin] PATH_INFO: `/v3/projects/1b1ee6f6a3c3475e8ae993c79ace9166/users/cca36a277e3643a498b27e3c5d5e14f4/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:49.119004 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbe35385-f120-4e14-b1c8-42092b7fde10 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1b1ee6f6a3c3475e8ae993c79ace9166, user_id=cca36a277e3643a498b27e3c5d5e14f4, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:49.121738 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbe35385-f120-4e14-b1c8-42092b7fde10 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:49.134787 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 646/1274] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:23:49 2022] PUT /identity/v3/projects/1b1ee6f6a3c3475e8ae993c79ace9166/users/cca36a277e3643a498b27e3c5d5e14f4/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:23:49.135726 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe35385-f120-4e14-b1c8-42092b7fde10 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:49.138982 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e716bb30-0814-4fbb-ac77-64ae7997e719 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:49.139733 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e716bb30-0814-4fbb-ac77-64ae7997e719 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:49.140032 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e716bb30-0814-4fbb-ac77-64ae7997e719 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:49.140300 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e716bb30-0814-4fbb-ac77-64ae7997e719 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:49.140940 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e716bb30-0814-4fbb-ac77-64ae7997e719 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:49.142602 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e716bb30-0814-4fbb-ac77-64ae7997e719 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:49.155844 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 629/1275] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:23:49 2022] GET /identity/v3/roles => generated 1420 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:23:49.155844 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e716bb30-0814-4fbb-ac77-64ae7997e719 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:49.163762 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e716bb30-0814-4fbb-ac77-64ae7997e719 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:49.202669 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe35385-f120-4e14-b1c8-42092b7fde10 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:49.207817 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85264bc3-0359-45a5-956c-a9cce3fd9ab5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:49.208421 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85264bc3-0359-45a5-956c-a9cce3fd9ab5 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:49.208701 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85264bc3-0359-45a5-956c-a9cce3fd9ab5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:49.208971 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85264bc3-0359-45a5-956c-a9cce3fd9ab5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:49.209648 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85264bc3-0359-45a5-956c-a9cce3fd9ab5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:49.211237 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85264bc3-0359-45a5-956c-a9cce3fd9ab5 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:49.219847 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512b5bbd-c911-415f-a414-bbb809e0d1fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:49.220492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512b5bbd-c911-415f-a414-bbb809e0d1fd admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:49.220766 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512b5bbd-c911-415f-a414-bbb809e0d1fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:49.220868 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 647/1276] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:23:49 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:23:49.221038 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512b5bbd-c911-415f-a414-bbb809e0d1fd admin admin] PATH_INFO: `/v3/projects/7eefe01b108148bab60d4de13be0405d/users/d311fc1f82124819b509f766fa432e2c/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:49.221649 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85264bc3-0359-45a5-956c-a9cce3fd9ab5 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:49.231639 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85264bc3-0359-45a5-956c-a9cce3fd9ab5 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:49.236424 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-512b5bbd-c911-415f-a414-bbb809e0d1fd admin admin] RBAC: Authorizing `identity:create_grant(project_id=7eefe01b108148bab60d4de13be0405d, user_id=d311fc1f82124819b509f766fa432e2c, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:49.236928 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf00834-89fe-4141-8446-4775b7afa5a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:49.237545 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf00834-89fe-4141-8446-4775b7afa5a3 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:49.237831 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf00834-89fe-4141-8446-4775b7afa5a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:49.238101 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf00834-89fe-4141-8446-4775b7afa5a3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:49.238961 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf00834-89fe-4141-8446-4775b7afa5a3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:49.239167 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-512b5bbd-c911-415f-a414-bbb809e0d1fd admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:49.240627 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf00834-89fe-4141-8446-4775b7afa5a3 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:49.251752 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 648/1277] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:23:49 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:23:49.252519 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf00834-89fe-4141-8446-4775b7afa5a3 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:49.256195 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 630/1278] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:23:49 2022] PUT /identity/v3/projects/7eefe01b108148bab60d4de13be0405d/users/d311fc1f82124819b509f766fa432e2c/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:23:49.257526 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e648f326-56b6-4dd2-b4b8-ff7fbf17a5eb None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:49.257814 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e648f326-56b6-4dd2-b4b8-ff7fbf17a5eb None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:49.258086 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e648f326-56b6-4dd2-b4b8-ff7fbf17a5eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:49.263467 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf00834-89fe-4141-8446-4775b7afa5a3 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:49.277042 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-e648f326-56b6-4dd2-b4b8-ff7fbf17a5eb None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:23:49.278251 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e648f326-56b6-4dd2-b4b8-ff7fbf17a5eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:49.311794 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:23:49.320332 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40fda65-bf36-41c5-b142-4ca37db9b88b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:49.321044 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40fda65-bf36-41c5-b142-4ca37db9b88b admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:49.321356 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40fda65-bf36-41c5-b142-4ca37db9b88b admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:49.321626 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40fda65-bf36-41c5-b142-4ca37db9b88b admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:49.322261 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40fda65-bf36-41c5-b142-4ca37db9b88b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:49.324165 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40fda65-bf36-41c5-b142-4ca37db9b88b admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:49.340136 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 631/1279] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:23:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:23:49.341392 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b39054-0a12-4039-9520-b783740b7605 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:49.341674 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b39054-0a12-4039-9520-b783740b7605 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:49.341941 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b39054-0a12-4039-9520-b783740b7605 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:49.358452 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 649/1280] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:23:49 2022] POST /identity/v3/projects => generated 387 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:23:49.358452 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc115643-4bf2-43b9-b0f2-176617ff2a68 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:49.358452 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc115643-4bf2-43b9-b0f2-176617ff2a68 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:49.358452 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc115643-4bf2-43b9-b0f2-176617ff2a68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:49.371568 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-e5b39054-0a12-4039-9520-b783740b7605 None None] MFA Rules not processed for user `931e14007f454ec9be4b44c39b3a9e22`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:23:49.372759 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e5b39054-0a12-4039-9520-b783740b7605 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:49.376056 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-dc115643-4bf2-43b9-b0f2-176617ff2a68 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:23:49.377250 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-dc115643-4bf2-43b9-b0f2-176617ff2a68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:49.393510 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:23:49.437256 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 632/1281] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:23:49 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:23:49.440281 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 650/1282] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:23:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:23:49.441188 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc115643-4bf2-43b9-b0f2-176617ff2a68 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:49.443439 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b39054-0a12-4039-9520-b783740b7605 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:49.453549 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc115643-4bf2-43b9-b0f2-176617ff2a68 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:49.459076 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-069f53b7-e38c-4c7d-8409-948ecb5c99bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:49.460065 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069f53b7-e38c-4c7d-8409-948ecb5c99bc admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:49.460895 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069f53b7-e38c-4c7d-8409-948ecb5c99bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:49.461299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b39054-0a12-4039-9520-b783740b7605 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:49.461557 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069f53b7-e38c-4c7d-8409-948ecb5c99bc admin admin] PATH_INFO: `/v3/projects/db70dacc270d431182451b57b277484f/users/99bcd22ff73b4526a54cf0e76f509436/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:49.466272 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa50d99-d6c1-46e4-8299-0df4fcde1339 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:49.466944 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa50d99-d6c1-46e4-8299-0df4fcde1339 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:49.467216 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa50d99-d6c1-46e4-8299-0df4fcde1339 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:49.467482 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa50d99-d6c1-46e4-8299-0df4fcde1339 admin admin] PATH_INFO: `/v3/projects/1dbe731cc92548a49f9ee0d6abd4ad3e/users/c57e9aa111044db39778f796f17b4ead/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:49.482560 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa50d99-d6c1-46e4-8299-0df4fcde1339 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1dbe731cc92548a49f9ee0d6abd4ad3e, user_id=c57e9aa111044db39778f796f17b4ead, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:49.483446 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-069f53b7-e38c-4c7d-8409-948ecb5c99bc admin admin] RBAC: Authorizing `identity:create_grant(project_id=db70dacc270d431182451b57b277484f, user_id=99bcd22ff73b4526a54cf0e76f509436, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:49.485311 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa50d99-d6c1-46e4-8299-0df4fcde1339 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:49.486232 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-069f53b7-e38c-4c7d-8409-948ecb5c99bc admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:49.499070 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 633/1283] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:23:49 2022] PUT /identity/v3/projects/1dbe731cc92548a49f9ee0d6abd4ad3e/users/c57e9aa111044db39778f796f17b4ead/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:23:49.509402 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa50d99-d6c1-46e4-8299-0df4fcde1339 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:49.509524 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 651/1284] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:23:49 2022] PUT /identity/v3/projects/db70dacc270d431182451b57b277484f/users/99bcd22ff73b4526a54cf0e76f509436/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:23:49.509524 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90baf9ac-5af4-4f4c-a301-a20526018e1d None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:49.509524 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90baf9ac-5af4-4f4c-a301-a20526018e1d None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:49.509524 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90baf9ac-5af4-4f4c-a301-a20526018e1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:49.521541 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-90baf9ac-5af4-4f4c-a301-a20526018e1d None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:23:49.522757 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-90baf9ac-5af4-4f4c-a301-a20526018e1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:49.570333 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa50d99-d6c1-46e4-8299-0df4fcde1339 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:49.583039 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 652/1285] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:23:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:23:49.587195 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90baf9ac-5af4-4f4c-a301-a20526018e1d None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:49.619221 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:23:49.630720 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5fc0614-ccc3-4db7-bcdb-2acc3946c033 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:49.631396 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fc0614-ccc3-4db7-bcdb-2acc3946c033 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:49.631671 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fc0614-ccc3-4db7-bcdb-2acc3946c033 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:49.631937 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fc0614-ccc3-4db7-bcdb-2acc3946c033 admin admin] PATH_INFO: `/v3/projects/53adc5168fae411d8b4e6d74693d4f92/users/d1df22d0d5bf4dd6b1bdfb561b9027b4/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:49.642228 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90baf9ac-5af4-4f4c-a301-a20526018e1d None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:49.646568 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646538c4-f639-42c6-a991-eadcd7bfe698 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:49.647051 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5fc0614-ccc3-4db7-bcdb-2acc3946c033 admin admin] RBAC: Authorizing `identity:create_grant(project_id=53adc5168fae411d8b4e6d74693d4f92, user_id=d1df22d0d5bf4dd6b1bdfb561b9027b4, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:49.647175 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646538c4-f639-42c6-a991-eadcd7bfe698 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:49.647434 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646538c4-f639-42c6-a991-eadcd7bfe698 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:49.647704 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646538c4-f639-42c6-a991-eadcd7bfe698 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:49.649969 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5fc0614-ccc3-4db7-bcdb-2acc3946c033 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:49.657956 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-646538c4-f639-42c6-a991-eadcd7bfe698 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:49.659428 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-646538c4-f639-42c6-a991-eadcd7bfe698 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:49.663484 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 634/1286] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:23:49 2022] PUT /identity/v3/projects/53adc5168fae411d8b4e6d74693d4f92/users/d1df22d0d5bf4dd6b1bdfb561b9027b4/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:23:49.664496 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5fc0614-ccc3-4db7-bcdb-2acc3946c033 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:49.669996 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 653/1287] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:23:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:23:49.672469 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646538c4-f639-42c6-a991-eadcd7bfe698 service neutron] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:49.682588 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646538c4-f639-42c6-a991-eadcd7bfe698 service neutron] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:49.731613 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:23:49.742480 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5fc0614-ccc3-4db7-bcdb-2acc3946c033 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:49.744922 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869a1538-c748-413a-b75b-cf2e065bdd02 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:49.745758 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869a1538-c748-413a-b75b-cf2e065bdd02 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:49.746444 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869a1538-c748-413a-b75b-cf2e065bdd02 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:49.746826 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869a1538-c748-413a-b75b-cf2e065bdd02 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:49.747756 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4255908-5575-46a3-ad2d-c33ef4c79fe7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:49.748571 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4255908-5575-46a3-ad2d-c33ef4c79fe7 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:49.748852 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4255908-5575-46a3-ad2d-c33ef4c79fe7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:49.749158 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4255908-5575-46a3-ad2d-c33ef4c79fe7 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:49.749789 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4255908-5575-46a3-ad2d-c33ef4c79fe7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:49.751713 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4255908-5575-46a3-ad2d-c33ef4c79fe7 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:49.759183 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-869a1538-c748-413a-b75b-cf2e065bdd02 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:49.760693 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-869a1538-c748-413a-b75b-cf2e065bdd02 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:49.772142 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 654/1288] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:23:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:23:49.773430 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093cf40a-38eb-4392-998a-2f2f465b845c None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:49.773761 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093cf40a-38eb-4392-998a-2f2f465b845c None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:49.773978 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093cf40a-38eb-4392-998a-2f2f465b845c None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:49.775742 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 655/1289] 10.222.0.42 () {54 vars in 961 bytes} [Thu Oct 27 01:23:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:23:49.776775 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb86b5a-e53b-48b4-86ac-5ea7c0c22067 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:49.777045 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb86b5a-e53b-48b4-86ac-5ea7c0c22067 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:49.777204 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 635/1290] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:23:49 2022] POST /identity/v3/users => generated 300 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:23:49.777353 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb86b5a-e53b-48b4-86ac-5ea7c0c22067 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:49.778833 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb9f76e-793f-4d36-b492-51d74843c7b1 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:49.779190 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb9f76e-793f-4d36-b492-51d74843c7b1 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:49.779534 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb9f76e-793f-4d36-b492-51d74843c7b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:49.797544 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-8cb86b5a-e53b-48b4-86ac-5ea7c0c22067 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:23:49.802691 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-8cb86b5a-e53b-48b4-86ac-5ea7c0c22067 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:49.820914 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-abb9f76e-793f-4d36-b492-51d74843c7b1 None None] MFA Rules not processed for user `c57e9aa111044db39778f796f17b4ead`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:23:49.822485 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-abb9f76e-793f-4d36-b492-51d74843c7b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:49.827743 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 656/1291] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:23:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:23:49.828938 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb86b5a-e53b-48b4-86ac-5ea7c0c22067 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:49.830456 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-8cb86b5a-e53b-48b4-86ac-5ea7c0c22067 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:49.844778 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb86b5a-e53b-48b4-86ac-5ea7c0c22067 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:49.850075 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84380a80-4860-4123-b453-66659470126e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:49.850736 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84380a80-4860-4123-b453-66659470126e service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:49.851026 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84380a80-4860-4123-b453-66659470126e service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:49.851337 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84380a80-4860-4123-b453-66659470126e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:49.862296 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84380a80-4860-4123-b453-66659470126e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:49.864230 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84380a80-4860-4123-b453-66659470126e service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:49.875505 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 657/1292] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:23:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:23:49.876921 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84cd6cd6-0091-4fc0-a926-7d5a0545a9b0 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:49.877248 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84cd6cd6-0091-4fc0-a926-7d5a0545a9b0 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:49.877523 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84cd6cd6-0091-4fc0-a926-7d5a0545a9b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:49.911480 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-84cd6cd6-0091-4fc0-a926-7d5a0545a9b0 None None] MFA Rules not processed for user `d1df22d0d5bf4dd6b1bdfb561b9027b4`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:23:49.912936 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-84cd6cd6-0091-4fc0-a926-7d5a0545a9b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:49.918873 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 636/1293] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:23:49 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:23:49.920248 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb9f76e-793f-4d36-b492-51d74843c7b1 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:49.943284 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb9f76e-793f-4d36-b492-51d74843c7b1 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:49.949155 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0f525a-13db-494c-baa6-9947b1d90955 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:49.949888 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0f525a-13db-494c-baa6-9947b1d90955 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:49.950366 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0f525a-13db-494c-baa6-9947b1d90955 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:49.950647 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0f525a-13db-494c-baa6-9947b1d90955 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:49.951456 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f0f525a-13db-494c-baa6-9947b1d90955 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:49.953572 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f0f525a-13db-494c-baa6-9947b1d90955 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:49.968834 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 637/1294] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:23:49 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:23:49.970506 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec07654-61db-4b04-9bf6-46f7c1f697ef None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:49.970977 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec07654-61db-4b04-9bf6-46f7c1f697ef None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:49.971259 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec07654-61db-4b04-9bf6-46f7c1f697ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:50.012290 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 658/1295] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:23:49 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:23:50.013692 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84cd6cd6-0091-4fc0-a926-7d5a0545a9b0 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:50.033025 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84cd6cd6-0091-4fc0-a926-7d5a0545a9b0 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:50.035966 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-6ec07654-61db-4b04-9bf6-46f7c1f697ef None None] MFA Rules not processed for user `51033d56f85f4282a49ded6338440677`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:23:50.038213 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-6ec07654-61db-4b04-9bf6-46f7c1f697ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:50.044407 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e04265-0506-46ec-bb61-4f345815eb64 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:50.045687 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e04265-0506-46ec-bb61-4f345815eb64 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:50.046275 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e04265-0506-46ec-bb61-4f345815eb64 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:50.047700 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e04265-0506-46ec-bb61-4f345815eb64 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:50.062597 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43e04265-0506-46ec-bb61-4f345815eb64 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:50.064304 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43e04265-0506-46ec-bb61-4f345815eb64 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:50.086109 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 659/1296] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:23:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 72 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:23:50.087289 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e04265-0506-46ec-bb61-4f345815eb64 service neutron] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:50.096290 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:23:50.182843 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 638/1297] 10.222.0.42 () {58 vars in 1058 bytes} [Thu Oct 27 01:23:49 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:23:50.183969 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec07654-61db-4b04-9bf6-46f7c1f697ef None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:50.206509 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec07654-61db-4b04-9bf6-46f7c1f697ef None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:50.212812 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e04265-0506-46ec-bb61-4f345815eb64 service neutron] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:50.215822 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad02b18-33c4-48e5-a98f-b4ad8bd0203a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:50.216621 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad02b18-33c4-48e5-a98f-b4ad8bd0203a admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:50.217524 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad02b18-33c4-48e5-a98f-b4ad8bd0203a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:50.218299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad02b18-33c4-48e5-a98f-b4ad8bd0203a admin admin] PATH_INFO: `/v3/projects/cf5c1960db2644a99cca6b1a76b86d0d/users/f2f8c8d5201b4945bc10956a81c49097/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:50.218890 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f20b5506-a383-4031-a779-137cf8b01a3c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:50.219649 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20b5506-a383-4031-a779-137cf8b01a3c service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:50.219941 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20b5506-a383-4031-a779-137cf8b01a3c service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:50.220213 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20b5506-a383-4031-a779-137cf8b01a3c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:50.257926 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad02b18-33c4-48e5-a98f-b4ad8bd0203a admin admin] RBAC: Authorizing `identity:create_grant(project_id=cf5c1960db2644a99cca6b1a76b86d0d, user_id=f2f8c8d5201b4945bc10956a81c49097, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:50.261692 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad02b18-33c4-48e5-a98f-b4ad8bd0203a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:50.263075 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f20b5506-a383-4031-a779-137cf8b01a3c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:50.264747 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f20b5506-a383-4031-a779-137cf8b01a3c service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:50.291237 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 639/1298] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:23:50 2022] PUT /identity/v3/projects/cf5c1960db2644a99cca6b1a76b86d0d/users/f2f8c8d5201b4945bc10956a81c49097/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:23:50.291723 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 660/1299] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:23:50 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 205 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:23:50.292374 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad02b18-33c4-48e5-a98f-b4ad8bd0203a admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:50.293092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f20b5506-a383-4031-a779-137cf8b01a3c service cinder] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:50.315575 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f20b5506-a383-4031-a779-137cf8b01a3c service cinder] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:50.429971 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:23:50.440830 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad02b18-33c4-48e5-a98f-b4ad8bd0203a admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:50.455668 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff52f06c-8d1d-4ef4-8ab3-aea84542464d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:50.456305 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff52f06c-8d1d-4ef4-8ab3-aea84542464d admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:50.456619 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff52f06c-8d1d-4ef4-8ab3-aea84542464d admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:50.456924 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff52f06c-8d1d-4ef4-8ab3-aea84542464d admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:50.457597 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff52f06c-8d1d-4ef4-8ab3-aea84542464d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:50.463316 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff52f06c-8d1d-4ef4-8ab3-aea84542464d admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:50.485866 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae572f59-7497-4617-b244-8f28a58b3aab service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:50.486672 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae572f59-7497-4617-b244-8f28a58b3aab service nova] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:50.487511 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae572f59-7497-4617-b244-8f28a58b3aab service nova] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:50.487638 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae572f59-7497-4617-b244-8f28a58b3aab service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:50.527079 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 640/1300] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:23:50 2022] POST /identity/v3/projects => generated 379 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:23:50.527079 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff52f06c-8d1d-4ef4-8ab3-aea84542464d admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:50.624544 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae572f59-7497-4617-b244-8f28a58b3aab service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:50.626219 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae572f59-7497-4617-b244-8f28a58b3aab service nova] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:50.640263 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 661/1301] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:23:50 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:23:50.641362 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae572f59-7497-4617-b244-8f28a58b3aab service nova] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:50.657257 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae572f59-7497-4617-b244-8f28a58b3aab service nova] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:50.661823 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff52f06c-8d1d-4ef4-8ab3-aea84542464d admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:50.665467 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3411698b-2fee-40b8-ae28-e45e72bfad2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:50.666092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3411698b-2fee-40b8-ae28-e45e72bfad2e admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:50.666376 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3411698b-2fee-40b8-ae28-e45e72bfad2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:50.666652 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3411698b-2fee-40b8-ae28-e45e72bfad2e admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:50.669730 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3411698b-2fee-40b8-ae28-e45e72bfad2e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:50.675448 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3411698b-2fee-40b8-ae28-e45e72bfad2e admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:50.676849 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0038e225-5d4c-4907-acef-ba67668547d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:50.677836 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0038e225-5d4c-4907-acef-ba67668547d7 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:50.678371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0038e225-5d4c-4907-acef-ba67668547d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:50.678863 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0038e225-5d4c-4907-acef-ba67668547d7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:50.679746 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0038e225-5d4c-4907-acef-ba67668547d7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:50.681880 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0038e225-5d4c-4907-acef-ba67668547d7 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:50.704433 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 641/1302] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:23:50 2022] GET /identity/v3/roles => generated 1420 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:23:50.709195 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0038e225-5d4c-4907-acef-ba67668547d7 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:50.728439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 662/1303] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:23:50 2022] POST /identity/v3/users => generated 296 bytes in 88 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:23:50.731819 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3411698b-2fee-40b8-ae28-e45e72bfad2e admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:50.732786 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0038e225-5d4c-4907-acef-ba67668547d7 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:50.748861 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c7a00d-416b-4611-9131-1a4fb341dd95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:50.749681 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c7a00d-416b-4611-9131-1a4fb341dd95 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:50.750068 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c7a00d-416b-4611-9131-1a4fb341dd95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:50.750446 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c7a00d-416b-4611-9131-1a4fb341dd95 admin admin] PATH_INFO: `/v3/projects/cf5c1960db2644a99cca6b1a76b86d0d/users/f2f8c8d5201b4945bc10956a81c49097/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:50.759123 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3411698b-2fee-40b8-ae28-e45e72bfad2e admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:50.760919 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c7a00d-416b-4611-9131-1a4fb341dd95 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cf5c1960db2644a99cca6b1a76b86d0d, user_id=f2f8c8d5201b4945bc10956a81c49097, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:50.764544 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c7a00d-416b-4611-9131-1a4fb341dd95 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:50.769940 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fec8d11-baf0-461e-993d-541184757538 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:50.774943 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fec8d11-baf0-461e-993d-541184757538 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:50.780168 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fec8d11-baf0-461e-993d-541184757538 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:50.780678 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fec8d11-baf0-461e-993d-541184757538 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:50.781456 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fec8d11-baf0-461e-993d-541184757538 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:50.783399 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fec8d11-baf0-461e-993d-541184757538 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:50.791811 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 642/1304] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:23:50 2022] PUT /identity/v3/projects/cf5c1960db2644a99cca6b1a76b86d0d/users/f2f8c8d5201b4945bc10956a81c49097/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:23:50.807989 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9950ad-d6cc-44e8-9916-cae9b277aba9 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:50.807989 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9950ad-d6cc-44e8-9916-cae9b277aba9 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:50.808397 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9950ad-d6cc-44e8-9916-cae9b277aba9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:50.818993 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 663/1305] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:23:50 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:23:50.824638 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fec8d11-baf0-461e-993d-541184757538 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:50.828397 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:23:50.856141 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-da9950ad-d6cc-44e8-9916-cae9b277aba9 None None] MFA Rules not processed for user `f2f8c8d5201b4945bc10956a81c49097`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:23:50.858737 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-da9950ad-d6cc-44e8-9916-cae9b277aba9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:50.862385 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fec8d11-baf0-461e-993d-541184757538 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:51.002504 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 643/1306] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:23:50 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:23:51.042095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9950ad-d6cc-44e8-9916-cae9b277aba9 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:51.050790 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b16156-3715-484b-aec2-deca75e10762 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:51.053698 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b16156-3715-484b-aec2-deca75e10762 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:51.053994 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b16156-3715-484b-aec2-deca75e10762 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:51.057095 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b16156-3715-484b-aec2-deca75e10762 admin admin] PATH_INFO: `/v3/projects/3fe71b3133b748c5b7b2c6442cc98a58/users/f9049e2900a94a489498b2105f45c8ef/roles/414286927fdd44688107ff5e10235acf` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:51.098736 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4b16156-3715-484b-aec2-deca75e10762 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3fe71b3133b748c5b7b2c6442cc98a58, user_id=f9049e2900a94a489498b2105f45c8ef, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:51.106838 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4b16156-3715-484b-aec2-deca75e10762 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:51.143915 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 664/1307] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:23:50 2022] PUT /identity/v3/projects/3fe71b3133b748c5b7b2c6442cc98a58/users/f9049e2900a94a489498b2105f45c8ef/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 318 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:23:51.146209 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b16156-3715-484b-aec2-deca75e10762 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:51.162564 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9950ad-d6cc-44e8-9916-cae9b277aba9 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:51.253308 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:23:51.281905 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df64622d-cfb5-42bb-862a-400b9c63a504 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:51.282739 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df64622d-cfb5-42bb-862a-400b9c63a504 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:51.283702 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df64622d-cfb5-42bb-862a-400b9c63a504 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:51.283702 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df64622d-cfb5-42bb-862a-400b9c63a504 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:51.319257 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b16156-3715-484b-aec2-deca75e10762 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:51.327039 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f92481-a355-4520-998a-231ccf77345d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:51.328332 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f92481-a355-4520-998a-231ccf77345d admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:51.329386 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f92481-a355-4520-998a-231ccf77345d admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:51.329844 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f92481-a355-4520-998a-231ccf77345d admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:51.331650 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f92481-a355-4520-998a-231ccf77345d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:51.336228 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f92481-a355-4520-998a-231ccf77345d admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:51.371716 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 665/1308] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:23:51 2022] GET /identity/v3/roles => generated 1420 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 27 01:23:51.373234 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11c8bee-0078-491d-b8c0-19ffea450817 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:51.373524 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11c8bee-0078-491d-b8c0-19ffea450817 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:51.373803 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11c8bee-0078-491d-b8c0-19ffea450817 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:51.375274 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 666/1309] 10.222.0.42 () {54 vars in 912 bytes} [Thu Oct 27 01:23:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:23:51.382126 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11c8bee-0078-491d-b8c0-19ffea450817 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:51.415970 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11c8bee-0078-491d-b8c0-19ffea450817 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:51.426833 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80577bca-d9bb-4f45-9438-767b9fbae74b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:51.428910 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80577bca-d9bb-4f45-9438-767b9fbae74b admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:51.430653 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:23:51.431157 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80577bca-d9bb-4f45-9438-767b9fbae74b admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:51.432392 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80577bca-d9bb-4f45-9438-767b9fbae74b admin admin] PATH_INFO: `/v3/projects/3fe71b3133b748c5b7b2c6442cc98a58/users/f9049e2900a94a489498b2105f45c8ef/roles/414286927fdd44688107ff5e10235acf` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:51.436561 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80577bca-d9bb-4f45-9438-767b9fbae74b admin admin] RBAC: Authorizing `identity:create_grant(project_id=3fe71b3133b748c5b7b2c6442cc98a58, user_id=f9049e2900a94a489498b2105f45c8ef, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:51.438332 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df64622d-cfb5-42bb-862a-400b9c63a504 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:51.439921 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df64622d-cfb5-42bb-862a-400b9c63a504 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:51.448743 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80577bca-d9bb-4f45-9438-767b9fbae74b admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:51.460486 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 644/1310] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:23:51 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 418 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:23:51.463246 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9b859f-210c-4d12-9a53-a7f09831f56c None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:51.464590 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9b859f-210c-4d12-9a53-a7f09831f56c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:51.465017 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9b859f-210c-4d12-9a53-a7f09831f56c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:51.501821 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 667/1311] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:23:51 2022] PUT /identity/v3/projects/3fe71b3133b748c5b7b2c6442cc98a58/users/f9049e2900a94a489498b2105f45c8ef/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:23:51.506319 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1963d164-12f5-4621-9c77-f3ec1d300b35 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:51.506747 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1963d164-12f5-4621-9c77-f3ec1d300b35 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:51.507141 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1963d164-12f5-4621-9c77-f3ec1d300b35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:51.533693 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-3e9b859f-210c-4d12-9a53-a7f09831f56c None None] MFA Rules not processed for user `51033d56f85f4282a49ded6338440677`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:23:51.534929 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-3e9b859f-210c-4d12-9a53-a7f09831f56c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:51.598545 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-1963d164-12f5-4621-9c77-f3ec1d300b35 None None] MFA Rules not processed for user `f9049e2900a94a489498b2105f45c8ef`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:23:51.604289 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-1963d164-12f5-4621-9c77-f3ec1d300b35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:51.712227 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 645/1312] 10.222.0.42 () {58 vars in 1009 bytes} [Thu Oct 27 01:23:51 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 249 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:23:51.725100 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9b859f-210c-4d12-9a53-a7f09831f56c None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:51.759738 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 668/1313] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:23:51 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 255 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:23:51.797002 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1963d164-12f5-4621-9c77-f3ec1d300b35 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:51.858835 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9b859f-210c-4d12-9a53-a7f09831f56c None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:51.869732 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ff216c-5a11-48d6-ad11-f764b1abffa9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:51.870321 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ff216c-5a11-48d6-ad11-f764b1abffa9 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:51.870595 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ff216c-5a11-48d6-ad11-f764b1abffa9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:51.874205 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ff216c-5a11-48d6-ad11-f764b1abffa9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:51.899509 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2ff216c-5a11-48d6-ad11-f764b1abffa9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:51.901740 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2ff216c-5a11-48d6-ad11-f764b1abffa9 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:51.924301 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 646/1314] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:23:51 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 200 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 27 01:23:51.925767 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ff216c-5a11-48d6-ad11-f764b1abffa9 service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:51.947085 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1963d164-12f5-4621-9c77-f3ec1d300b35 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:51.950544 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ff216c-5a11-48d6-ad11-f764b1abffa9 service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:51.957913 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4f1414-58d5-4ca6-9a2b-f33a91c94749 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:51.960195 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4f1414-58d5-4ca6-9a2b-f33a91c94749 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:51.960485 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4f1414-58d5-4ca6-9a2b-f33a91c94749 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:51.960848 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4f1414-58d5-4ca6-9a2b-f33a91c94749 admin admin] PATH_INFO: `/v3/users/26d5719e31114f55abdd4b3f1acacdde` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:51.962065 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b86ca16b-b8d2-49bf-a056-90d4aef98d16 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:51.963479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e4f1414-58d5-4ca6-9a2b-f33a91c94749 admin admin] RBAC: Authorizing `identity:delete_user(user_id=26d5719e31114f55abdd4b3f1acacdde)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:51.963839 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86ca16b-b8d2-49bf-a056-90d4aef98d16 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:51.964132 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86ca16b-b8d2-49bf-a056-90d4aef98d16 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:51.964395 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86ca16b-b8d2-49bf-a056-90d4aef98d16 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:51.968516 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e4f1414-58d5-4ca6-9a2b-f33a91c94749 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:51.992339 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b86ca16b-b8d2-49bf-a056-90d4aef98d16 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:51.994785 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b86ca16b-b8d2-49bf-a056-90d4aef98d16 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:52.032572 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 647/1315] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:23:51 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:23:52.136188 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=92,keystone:INSERT=1,keystone:DELETE=7 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:23:52.144247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-4e4f1414-58d5-4ca6-9a2b-f33a91c94749 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '26d5719e31114f55abdd4b3f1acacdde'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:52.168447 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-4e4f1414-58d5-4ca6-9a2b-f33a91c94749 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '26d5719e31114f55abdd4b3f1acacdde'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:52.206291 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-4e4f1414-58d5-4ca6-9a2b-f33a91c94749 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '26d5719e31114f55abdd4b3f1acacdde'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:52.207071 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-4e4f1414-58d5-4ca6-9a2b-f33a91c94749 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '26d5719e31114f55abdd4b3f1acacdde'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:52.212282 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-4e4f1414-58d5-4ca6-9a2b-f33a91c94749 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '26d5719e31114f55abdd4b3f1acacdde'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:52.249667 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 669/1316] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:23:51 2022] DELETE /identity/v3/users/26d5719e31114f55abdd4b3f1acacdde => generated 0 bytes in 450 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:23:52.255423 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97c9255-ea2b-419d-b064-dbc5994180dc None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:52.255712 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97c9255-ea2b-419d-b064-dbc5994180dc None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:52.255990 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97c9255-ea2b-419d-b064-dbc5994180dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:52.306874 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-d97c9255-ea2b-419d-b064-dbc5994180dc None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:23:52.308127 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d97c9255-ea2b-419d-b064-dbc5994180dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:52.483144 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 648/1317] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:23:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 229 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:23:52.596037 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4f1414-58d5-4ca6-9a2b-f33a91c94749 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:52.597678 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-4e4f1414-58d5-4ca6-9a2b-f33a91c94749 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:52.794092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4f1414-58d5-4ca6-9a2b-f33a91c94749 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:52.814086 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b9ff43-80e3-405f-b438-4bf03baaaaf2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:52.816989 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b9ff43-80e3-405f-b438-4bf03baaaaf2 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:52.817302 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b9ff43-80e3-405f-b438-4bf03baaaaf2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:52.817575 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b9ff43-80e3-405f-b438-4bf03baaaaf2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:52.854345 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b9ff43-80e3-405f-b438-4bf03baaaaf2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:52.858025 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b9ff43-80e3-405f-b438-4bf03baaaaf2 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:52.866151 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d97c9255-ea2b-419d-b064-dbc5994180dc None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:52.868496 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d97c9255-ea2b-419d-b064-dbc5994180dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:52.892673 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 670/1318] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:23:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 297 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:23:52.921544 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:23:53.014672 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d97c9255-ea2b-419d-b064-dbc5994180dc None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:53.024273 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e9a0c4-3bb5-4f73-8c1b-b4ac565ac33b tempest-VolumeQuotaClassesTest-783075546 tempest-VolumeQuotaClassesTest-783075546-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9049e2900a94a489498b2105f45c8ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3fe71b3133b748c5b7b2c6442cc98a58', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:53.026073 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e9a0c4-3bb5-4f73-8c1b-b4ac565ac33b tempest-VolumeQuotaClassesTest-783075546 tempest-VolumeQuotaClassesTest-783075546-project-admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:53.026675 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e9a0c4-3bb5-4f73-8c1b-b4ac565ac33b tempest-VolumeQuotaClassesTest-783075546 tempest-VolumeQuotaClassesTest-783075546-project-admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:53.028538 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e9a0c4-3bb5-4f73-8c1b-b4ac565ac33b tempest-VolumeQuotaClassesTest-783075546 tempest-VolumeQuotaClassesTest-783075546-project-admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:53.030471 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3e9a0c4-3bb5-4f73-8c1b-b4ac565ac33b tempest-VolumeQuotaClassesTest-783075546 tempest-VolumeQuotaClassesTest-783075546-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:23:53.030907 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3e9a0c4-3bb5-4f73-8c1b-b4ac565ac33b tempest-VolumeQuotaClassesTest-783075546 tempest-VolumeQuotaClassesTest-783075546-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:53.034273 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3e9a0c4-3bb5-4f73-8c1b-b4ac565ac33b tempest-VolumeQuotaClassesTest-783075546 tempest-VolumeQuotaClassesTest-783075546-project-admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:53.072902 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 649/1319] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:23:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:23:53.074106 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b9ff43-80e3-405f-b438-4bf03baaaaf2 service neutron] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:53.089825 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b9ff43-80e3-405f-b438-4bf03baaaaf2 service neutron] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:53.102255 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd3c4f1-e8b2-4e23-ae16-2006d1870e66 tempest-VolumeQuotaClassesTest-783075546 tempest-VolumeQuotaClassesTest-783075546-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9049e2900a94a489498b2105f45c8ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3fe71b3133b748c5b7b2c6442cc98a58', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:53.102884 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd3c4f1-e8b2-4e23-ae16-2006d1870e66 tempest-VolumeQuotaClassesTest-783075546 tempest-VolumeQuotaClassesTest-783075546-project-admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:53.103169 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd3c4f1-e8b2-4e23-ae16-2006d1870e66 tempest-VolumeQuotaClassesTest-783075546 tempest-VolumeQuotaClassesTest-783075546-project-admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:53.103447 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd3c4f1-e8b2-4e23-ae16-2006d1870e66 tempest-VolumeQuotaClassesTest-783075546 tempest-VolumeQuotaClassesTest-783075546-project-admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:53.104093 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd3c4f1-e8b2-4e23-ae16-2006d1870e66 tempest-VolumeQuotaClassesTest-783075546 tempest-VolumeQuotaClassesTest-783075546-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:53.106044 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd3c4f1-e8b2-4e23-ae16-2006d1870e66 tempest-VolumeQuotaClassesTest-783075546 tempest-VolumeQuotaClassesTest-783075546-project-admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:53.143648 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0865c5d5-99fb-4db5-a1c5-4787f5c633d4 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:53.143931 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0865c5d5-99fb-4db5-a1c5-4787f5c633d4 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:53.144230 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0865c5d5-99fb-4db5-a1c5-4787f5c633d4 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:53.146348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 650/1320] 10.222.0.42 () {54 vars in 961 bytes} [Thu Oct 27 01:23:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:23:53.155341 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8232dcb5-8774-4f62-b6b8-42c99cf5e49d None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:53.155629 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8232dcb5-8774-4f62-b6b8-42c99cf5e49d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:53.155932 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8232dcb5-8774-4f62-b6b8-42c99cf5e49d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:53.175589 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 671/1321] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:23:53 2022] POST /identity/v3/projects => generated 355 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:23:53.177812 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd3c4f1-e8b2-4e23-ae16-2006d1870e66 tempest-VolumeQuotaClassesTest-783075546 tempest-VolumeQuotaClassesTest-783075546-project-admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:53.194787 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-8232dcb5-8774-4f62-b6b8-42c99cf5e49d None None] MFA Rules not processed for user `51033d56f85f4282a49ded6338440677`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:23:53.196029 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-8232dcb5-8774-4f62-b6b8-42c99cf5e49d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:53.296275 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd3c4f1-e8b2-4e23-ae16-2006d1870e66 tempest-VolumeQuotaClassesTest-783075546 tempest-VolumeQuotaClassesTest-783075546-project-admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:53.313390 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4e4f9b-03bc-4067-af1e-220b4b7c1deb tempest-VolumeQuotaClassesTest-783075546 tempest-VolumeQuotaClassesTest-783075546-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9049e2900a94a489498b2105f45c8ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3fe71b3133b748c5b7b2c6442cc98a58', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:53.314031 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4e4f9b-03bc-4067-af1e-220b4b7c1deb tempest-VolumeQuotaClassesTest-783075546 tempest-VolumeQuotaClassesTest-783075546-project-admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:53.314316 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4e4f9b-03bc-4067-af1e-220b4b7c1deb tempest-VolumeQuotaClassesTest-783075546 tempest-VolumeQuotaClassesTest-783075546-project-admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:53.314596 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4e4f9b-03bc-4067-af1e-220b4b7c1deb tempest-VolumeQuotaClassesTest-783075546 tempest-VolumeQuotaClassesTest-783075546-project-admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:53.315262 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb4e4f9b-03bc-4067-af1e-220b4b7c1deb tempest-VolumeQuotaClassesTest-783075546 tempest-VolumeQuotaClassesTest-783075546-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:23:53.316904 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb4e4f9b-03bc-4067-af1e-220b4b7c1deb tempest-VolumeQuotaClassesTest-783075546 tempest-VolumeQuotaClassesTest-783075546-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:53.320833 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb4e4f9b-03bc-4067-af1e-220b4b7c1deb tempest-VolumeQuotaClassesTest-783075546 tempest-VolumeQuotaClassesTest-783075546-project-admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:53.341220 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 651/1322] 10.222.0.42 () {58 vars in 1058 bytes} [Thu Oct 27 01:23:53 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:23:53.349342 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 672/1323] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:23:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:23:53.351881 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f0c5bd-9d7f-4d55-b2d4-1b3f157c3c88 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:53.351881 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f0c5bd-9d7f-4d55-b2d4-1b3f157c3c88 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:53.352131 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f0c5bd-9d7f-4d55-b2d4-1b3f157c3c88 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:53.353673 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 652/1324] 10.222.0.42 () {54 vars in 912 bytes} [Thu Oct 27 01:23:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:23:53.374611 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb8577e-d89e-426f-bb2d-b8158825f2de None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:53.375021 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb8577e-d89e-426f-bb2d-b8158825f2de None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:53.375375 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb8577e-d89e-426f-bb2d-b8158825f2de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:53.413586 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-acb8577e-d89e-426f-bb2d-b8158825f2de None None] MFA Rules not processed for user `51033d56f85f4282a49ded6338440677`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:23:53.415458 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-acb8577e-d89e-426f-bb2d-b8158825f2de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:53.425169 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f0c5bd-9d7f-4d55-b2d4-1b3f157c3c88 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:53.432341 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 673/1325] 10.222.0.42 () {58 vars in 1009 bytes} [Thu Oct 27 01:23:53 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 59 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:23:53.450354 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f0c5bd-9d7f-4d55-b2d4-1b3f157c3c88 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:53.452849 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb8577e-d89e-426f-bb2d-b8158825f2de None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:53.454621 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-acb8577e-d89e-426f-bb2d-b8158825f2de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:53.464669 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-087230b0-1814-4b02-b896-26fce3cacec9 tempest-VolumeQuotaClassesTest-783075546 tempest-VolumeQuotaClassesTest-783075546-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9049e2900a94a489498b2105f45c8ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3fe71b3133b748c5b7b2c6442cc98a58', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:53.465355 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087230b0-1814-4b02-b896-26fce3cacec9 tempest-VolumeQuotaClassesTest-783075546 tempest-VolumeQuotaClassesTest-783075546-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:53.465642 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087230b0-1814-4b02-b896-26fce3cacec9 tempest-VolumeQuotaClassesTest-783075546 tempest-VolumeQuotaClassesTest-783075546-project-admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:53.465924 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087230b0-1814-4b02-b896-26fce3cacec9 tempest-VolumeQuotaClassesTest-783075546 tempest-VolumeQuotaClassesTest-783075546-project-admin] PATH_INFO: `/v3/projects/daedab6b0dd94f1ca282bf33f54abb70` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:53.467320 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-087230b0-1814-4b02-b896-26fce3cacec9 tempest-VolumeQuotaClassesTest-783075546 tempest-VolumeQuotaClassesTest-783075546-project-admin] RBAC: Authorizing `identity:delete_project(project_id=daedab6b0dd94f1ca282bf33f54abb70)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:53.469432 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-087230b0-1814-4b02-b896-26fce3cacec9 tempest-VolumeQuotaClassesTest-783075546 tempest-VolumeQuotaClassesTest-783075546-project-admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:53.515091 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=95,keystone:INSERT=2,keystone:DELETE=3 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:23:53.592529 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb8577e-d89e-426f-bb2d-b8158825f2de None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:53.613838 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-087230b0-1814-4b02-b896-26fce3cacec9 tempest-VolumeQuotaClassesTest-783075546 tempest-VolumeQuotaClassesTest-783075546-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'daedab6b0dd94f1ca282bf33f54abb70', 'request_id': 'req-087230b0-1814-4b02-b896-26fce3cacec9'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:53.633177 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-087230b0-1814-4b02-b896-26fce3cacec9 tempest-VolumeQuotaClassesTest-783075546 tempest-VolumeQuotaClassesTest-783075546-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'daedab6b0dd94f1ca282bf33f54abb70', 'request_id': 'req-087230b0-1814-4b02-b896-26fce3cacec9'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:53.716742 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-087230b0-1814-4b02-b896-26fce3cacec9 tempest-VolumeQuotaClassesTest-783075546 tempest-VolumeQuotaClassesTest-783075546-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'daedab6b0dd94f1ca282bf33f54abb70', 'request_id': 'req-087230b0-1814-4b02-b896-26fce3cacec9'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:53.716742 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-087230b0-1814-4b02-b896-26fce3cacec9 tempest-VolumeQuotaClassesTest-783075546 tempest-VolumeQuotaClassesTest-783075546-project-admin] The token cache is being invalidate because project daedab6b0dd94f1ca282bf33f54abb70 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:23:53.716742 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-087230b0-1814-4b02-b896-26fce3cacec9 tempest-VolumeQuotaClassesTest-783075546 tempest-VolumeQuotaClassesTest-783075546-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:53.716742 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-087230b0-1814-4b02-b896-26fce3cacec9 tempest-VolumeQuotaClassesTest-783075546 tempest-VolumeQuotaClassesTest-783075546-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:53.735220 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 653/1326] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:23:53 2022] DELETE /identity/v3/projects/daedab6b0dd94f1ca282bf33f54abb70 => generated 0 bytes in 289 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:23:53.774390 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca7ba6a-8988-4883-8532-afdfd683fb8d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:53.775955 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca7ba6a-8988-4883-8532-afdfd683fb8d service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:53.776511 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca7ba6a-8988-4883-8532-afdfd683fb8d service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:53.777094 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca7ba6a-8988-4883-8532-afdfd683fb8d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:53.784058 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-cca7ba6a-8988-4883-8532-afdfd683fb8d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:53.978089 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-087230b0-1814-4b02-b896-26fce3cacec9 tempest-VolumeQuotaClassesTest-783075546 tempest-VolumeQuotaClassesTest-783075546-project-admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:53.979750 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-087230b0-1814-4b02-b896-26fce3cacec9 tempest-VolumeQuotaClassesTest-783075546 tempest-VolumeQuotaClassesTest-783075546-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:54.030991 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca7ba6a-8988-4883-8532-afdfd683fb8d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:54.033643 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca7ba6a-8988-4883-8532-afdfd683fb8d service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:54.053588 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:23:54.084709 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 674/1327] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:23:53 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 632 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:23:54.142923 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-087230b0-1814-4b02-b896-26fce3cacec9 tempest-VolumeQuotaClassesTest-783075546 tempest-VolumeQuotaClassesTest-783075546-project-admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:54.155691 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7784e3cf-c373-449d-80a8-8adbf394593d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:54.155691 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7784e3cf-c373-449d-80a8-8adbf394593d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:54.155691 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7784e3cf-c373-449d-80a8-8adbf394593d admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:54.155691 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7784e3cf-c373-449d-80a8-8adbf394593d admin admin] PATH_INFO: `/v3/projects/612f14b893ea46228c14d87e0d843e99` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:54.157272 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7784e3cf-c373-449d-80a8-8adbf394593d admin admin] RBAC: Authorizing `identity:delete_project(project_id=612f14b893ea46228c14d87e0d843e99)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:54.159270 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7784e3cf-c373-449d-80a8-8adbf394593d admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:54.296763 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-7784e3cf-c373-449d-80a8-8adbf394593d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '612f14b893ea46228c14d87e0d843e99', 'request_id': 'req-7784e3cf-c373-449d-80a8-8adbf394593d'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:54.299261 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca7ba6a-8988-4883-8532-afdfd683fb8d service cinder] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:54.300724 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-cca7ba6a-8988-4883-8532-afdfd683fb8d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:54.326368 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-7784e3cf-c373-449d-80a8-8adbf394593d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '612f14b893ea46228c14d87e0d843e99', 'request_id': 'req-7784e3cf-c373-449d-80a8-8adbf394593d'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:54.332060 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca7ba6a-8988-4883-8532-afdfd683fb8d service cinder] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:54.339700 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-7784e3cf-c373-449d-80a8-8adbf394593d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '612f14b893ea46228c14d87e0d843e99', 'request_id': 'req-7784e3cf-c373-449d-80a8-8adbf394593d'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:54.378135 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-7784e3cf-c373-449d-80a8-8adbf394593d admin admin] The token cache is being invalidate because project 612f14b893ea46228c14d87e0d843e99 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:23:54.381603 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-7784e3cf-c373-449d-80a8-8adbf394593d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:54.384733 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-7784e3cf-c373-449d-80a8-8adbf394593d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:54.397647 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 654/1328] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:23:53 2022] DELETE /identity/v3/projects/612f14b893ea46228c14d87e0d843e99 => generated 0 bytes in 415 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:23:54.399639 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7784e3cf-c373-449d-80a8-8adbf394593d admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:54.401380 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-7784e3cf-c373-449d-80a8-8adbf394593d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:54.495590 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e2186b-fb6c-4f7a-8cce-1c6564142431 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:54.496740 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e2186b-fb6c-4f7a-8cce-1c6564142431 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:54.497379 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e2186b-fb6c-4f7a-8cce-1c6564142431 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:54.497882 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e2186b-fb6c-4f7a-8cce-1c6564142431 admin admin] PATH_INFO: `/v3/users/d1df22d0d5bf4dd6b1bdfb561b9027b4` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:54.501017 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87e2186b-fb6c-4f7a-8cce-1c6564142431 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d1df22d0d5bf4dd6b1bdfb561b9027b4)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:54.510232 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87e2186b-fb6c-4f7a-8cce-1c6564142431 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:54.588998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7784e3cf-c373-449d-80a8-8adbf394593d admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:54.600877 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:23:54.604692 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f157730-e422-4649-b9e5-7da00017a77e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:54.605615 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f157730-e422-4649-b9e5-7da00017a77e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:54.607508 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f157730-e422-4649-b9e5-7da00017a77e admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:54.608889 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f157730-e422-4649-b9e5-7da00017a77e admin admin] PATH_INFO: `/v3/users/f2f8c8d5201b4945bc10956a81c49097` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:54.613183 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f157730-e422-4649-b9e5-7da00017a77e admin admin] RBAC: Authorizing `identity:delete_user(user_id=f2f8c8d5201b4945bc10956a81c49097)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:54.617206 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: WARNING py.warnings [None req-5f157730-e422-4649-b9e5-7da00017a77e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 27 01:23:54.617206 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: warnings.warn(msg) Oct 27 01:23:54.617206 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: Oct 27 01:23:54.621407 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f157730-e422-4649-b9e5-7da00017a77e admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:54.639546 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-87e2186b-fb6c-4f7a-8cce-1c6564142431 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd1df22d0d5bf4dd6b1bdfb561b9027b4'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:54.670107 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-87e2186b-fb6c-4f7a-8cce-1c6564142431 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd1df22d0d5bf4dd6b1bdfb561b9027b4'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:54.698884 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-87e2186b-fb6c-4f7a-8cce-1c6564142431 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd1df22d0d5bf4dd6b1bdfb561b9027b4'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:54.701464 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-87e2186b-fb6c-4f7a-8cce-1c6564142431 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd1df22d0d5bf4dd6b1bdfb561b9027b4'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:54.702897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-87e2186b-fb6c-4f7a-8cce-1c6564142431 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd1df22d0d5bf4dd6b1bdfb561b9027b4'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:54.745286 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 675/1329] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:23:54 2022] DELETE /identity/v3/users/d1df22d0d5bf4dd6b1bdfb561b9027b4 => generated 0 bytes in 446 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:23:54.749985 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4fd3039-0ad5-4e68-a903-1e6150aab2d8 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:54.750445 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4fd3039-0ad5-4e68-a903-1e6150aab2d8 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:54.750728 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4fd3039-0ad5-4e68-a903-1e6150aab2d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:54.802107 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-5f157730-e422-4649-b9e5-7da00017a77e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f2f8c8d5201b4945bc10956a81c49097'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:54.828683 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-b4fd3039-0ad5-4e68-a903-1e6150aab2d8 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:23:54.829300 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-5f157730-e422-4649-b9e5-7da00017a77e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f2f8c8d5201b4945bc10956a81c49097'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:54.830031 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-5f157730-e422-4649-b9e5-7da00017a77e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f2f8c8d5201b4945bc10956a81c49097'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:54.839020 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-b4fd3039-0ad5-4e68-a903-1e6150aab2d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:54.867307 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:23:54.901569 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-5f157730-e422-4649-b9e5-7da00017a77e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f2f8c8d5201b4945bc10956a81c49097'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:54.922040 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-5f157730-e422-4649-b9e5-7da00017a77e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f2f8c8d5201b4945bc10956a81c49097'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:54.938571 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 655/1330] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:23:54 2022] DELETE /identity/v3/users/f2f8c8d5201b4945bc10956a81c49097 => generated 0 bytes in 539 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:23:55.051005 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 676/1331] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:23:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 302 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:23:55.190257 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f157730-e422-4649-b9e5-7da00017a77e admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:55.193214 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-5f157730-e422-4649-b9e5-7da00017a77e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:55.361278 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f157730-e422-4649-b9e5-7da00017a77e admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:55.367013 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840c3005-0dae-400d-8ab9-2217abc88c3e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:55.367691 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840c3005-0dae-400d-8ab9-2217abc88c3e service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:55.368225 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840c3005-0dae-400d-8ab9-2217abc88c3e service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:55.368225 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840c3005-0dae-400d-8ab9-2217abc88c3e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:55.552810 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840c3005-0dae-400d-8ab9-2217abc88c3e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:55.557492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840c3005-0dae-400d-8ab9-2217abc88c3e service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:55.580915 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 656/1332] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:23:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:23:56.082145 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4fd3039-0ad5-4e68-a903-1e6150aab2d8 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:56.084033 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-b4fd3039-0ad5-4e68-a903-1e6150aab2d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:56.104892 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4fd3039-0ad5-4e68-a903-1e6150aab2d8 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:56.113123 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d17b7f-0a58-48fc-b994-393ee0be3097 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:56.114132 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d17b7f-0a58-48fc-b994-393ee0be3097 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:56.114650 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d17b7f-0a58-48fc-b994-393ee0be3097 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:56.115144 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d17b7f-0a58-48fc-b994-393ee0be3097 admin admin] PATH_INFO: `/v3/projects/cf5c1960db2644a99cca6b1a76b86d0d` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:56.117143 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d17b7f-0a58-48fc-b994-393ee0be3097 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cf5c1960db2644a99cca6b1a76b86d0d)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:56.118709 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: WARNING py.warnings [None req-75d17b7f-0a58-48fc-b994-393ee0be3097 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 27 01:23:56.118709 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: warnings.warn(msg) Oct 27 01:23:56.118709 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: Oct 27 01:23:56.121340 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d17b7f-0a58-48fc-b994-393ee0be3097 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:56.229960 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-75d17b7f-0a58-48fc-b994-393ee0be3097 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cf5c1960db2644a99cca6b1a76b86d0d', 'request_id': 'req-75d17b7f-0a58-48fc-b994-393ee0be3097'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:56.248676 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-75d17b7f-0a58-48fc-b994-393ee0be3097 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cf5c1960db2644a99cca6b1a76b86d0d', 'request_id': 'req-75d17b7f-0a58-48fc-b994-393ee0be3097'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:56.278300 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-75d17b7f-0a58-48fc-b994-393ee0be3097 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cf5c1960db2644a99cca6b1a76b86d0d', 'request_id': 'req-75d17b7f-0a58-48fc-b994-393ee0be3097'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:56.312292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-75d17b7f-0a58-48fc-b994-393ee0be3097 admin admin] The token cache is being invalidate because project cf5c1960db2644a99cca6b1a76b86d0d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:23:56.312599 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-75d17b7f-0a58-48fc-b994-393ee0be3097 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:56.314083 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-75d17b7f-0a58-48fc-b994-393ee0be3097 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:56.321504 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 677/1333] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:23:56 2022] DELETE /identity/v3/projects/cf5c1960db2644a99cca6b1a76b86d0d => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:23:56.326354 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ba9541-d8f4-447f-b6a9-6781fbe673d7 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:56.326640 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ba9541-d8f4-447f-b6a9-6781fbe673d7 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:56.327346 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ba9541-d8f4-447f-b6a9-6781fbe673d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:56.366256 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-07ba9541-d8f4-447f-b6a9-6781fbe673d7 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:23:56.367535 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-07ba9541-d8f4-447f-b6a9-6781fbe673d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:56.512193 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 657/1334] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:23:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:23:56.521892 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7633fe-874c-45af-8550-85d6eed60b84 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:56.522186 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7633fe-874c-45af-8550-85d6eed60b84 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:56.522458 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7633fe-874c-45af-8550-85d6eed60b84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:56.564327 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-be7633fe-874c-45af-8550-85d6eed60b84 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:23:56.572136 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-be7633fe-874c-45af-8550-85d6eed60b84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:56.593580 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 678/1335] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:23:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 73 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:23:56.598216 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ba9541-d8f4-447f-b6a9-6781fbe673d7 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:56.613434 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ba9541-d8f4-447f-b6a9-6781fbe673d7 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:56.628703 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1914fe-9539-4841-9d54-8cc763e13c8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:56.630109 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1914fe-9539-4841-9d54-8cc763e13c8f admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:56.630561 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1914fe-9539-4841-9d54-8cc763e13c8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:56.630995 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1914fe-9539-4841-9d54-8cc763e13c8f admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:56.631821 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd1914fe-9539-4841-9d54-8cc763e13c8f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:23:56.633092 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd1914fe-9539-4841-9d54-8cc763e13c8f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:56.635618 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd1914fe-9539-4841-9d54-8cc763e13c8f admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:56.673614 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 658/1336] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:23:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:23:56.679420 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7633fe-874c-45af-8550-85d6eed60b84 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:56.692295 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7633fe-874c-45af-8550-85d6eed60b84 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:56.705610 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6656c963-1094-4c98-b5f6-c5dd62319d64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:56.706456 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6656c963-1094-4c98-b5f6-c5dd62319d64 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:56.706592 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6656c963-1094-4c98-b5f6-c5dd62319d64 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:56.706761 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6656c963-1094-4c98-b5f6-c5dd62319d64 admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:56.707519 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6656c963-1094-4c98-b5f6-c5dd62319d64 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:56.711650 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6656c963-1094-4c98-b5f6-c5dd62319d64 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:56.782406 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 679/1337] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:23:56 2022] POST /identity/v3/projects => generated 383 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:23:56.783558 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1914fe-9539-4841-9d54-8cc763e13c8f admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:56.805515 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1914fe-9539-4841-9d54-8cc763e13c8f admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:56.822944 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:DELETE=2,keystone:SELECT=98 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:23:56.898476 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de0c163-151e-4ad2-b25f-082a0038ed3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:56.899553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de0c163-151e-4ad2-b25f-082a0038ed3f admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:56.900046 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de0c163-151e-4ad2-b25f-082a0038ed3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:56.900529 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de0c163-151e-4ad2-b25f-082a0038ed3f admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:56.902341 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de0c163-151e-4ad2-b25f-082a0038ed3f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:56.908635 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de0c163-151e-4ad2-b25f-082a0038ed3f admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:56.940337 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 659/1338] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:23:56 2022] POST /identity/v3/users => generated 292 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:23:56.944534 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6656c963-1094-4c98-b5f6-c5dd62319d64 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:56.956478 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:23:56.979485 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6656c963-1094-4c98-b5f6-c5dd62319d64 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:56.995590 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023727c5-cbdc-4361-ac50-6799e75a0cdd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:56.996602 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023727c5-cbdc-4361-ac50-6799e75a0cdd admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:56.998792 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023727c5-cbdc-4361-ac50-6799e75a0cdd admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:56.999859 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023727c5-cbdc-4361-ac50-6799e75a0cdd admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:57.005721 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023727c5-cbdc-4361-ac50-6799e75a0cdd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:57.008704 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023727c5-cbdc-4361-ac50-6799e75a0cdd admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:57.054078 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de0c163-151e-4ad2-b25f-082a0038ed3f admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:57.054586 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 680/1339] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:23:56 2022] GET /identity/v3/roles => generated 1420 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:23:57.083142 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de0c163-151e-4ad2-b25f-082a0038ed3f admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:57.095461 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d8f6e0-dcea-4898-ada9-1134e7dbd37e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:57.096480 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d8f6e0-dcea-4898-ada9-1134e7dbd37e admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:57.099655 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d8f6e0-dcea-4898-ada9-1134e7dbd37e admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:57.100304 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d8f6e0-dcea-4898-ada9-1134e7dbd37e admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:57.103376 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7d8f6e0-dcea-4898-ada9-1134e7dbd37e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:57.114373 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7d8f6e0-dcea-4898-ada9-1134e7dbd37e admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:57.154915 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 660/1340] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:23:57 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:23:57.155238 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023727c5-cbdc-4361-ac50-6799e75a0cdd admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:57.173042 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023727c5-cbdc-4361-ac50-6799e75a0cdd admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:57.204218 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce295a06-85c7-422b-87f1-47910e0e7f20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:57.205048 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce295a06-85c7-422b-87f1-47910e0e7f20 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:57.205560 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce295a06-85c7-422b-87f1-47910e0e7f20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:57.205943 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce295a06-85c7-422b-87f1-47910e0e7f20 admin admin] PATH_INFO: `/v3/projects/2d929d321b214af7977d4bbc558b86d1/users/5dad8decd91342cca9d4672f453dfff2/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:57.244029 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce295a06-85c7-422b-87f1-47910e0e7f20 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2d929d321b214af7977d4bbc558b86d1, user_id=5dad8decd91342cca9d4672f453dfff2, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:57.247496 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce295a06-85c7-422b-87f1-47910e0e7f20 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:57.293546 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 681/1341] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:23:57 2022] PUT /identity/v3/projects/2d929d321b214af7977d4bbc558b86d1/users/5dad8decd91342cca9d4672f453dfff2/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:23:57.304271 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ba311f-60e8-4103-b1c8-60861eed7006 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:57.304560 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ba311f-60e8-4103-b1c8-60861eed7006 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:57.304840 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ba311f-60e8-4103-b1c8-60861eed7006 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:57.370637 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-58ba311f-60e8-4103-b1c8-60861eed7006 None None] MFA Rules not processed for user `5dad8decd91342cca9d4672f453dfff2`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:23:57.374202 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-58ba311f-60e8-4103-b1c8-60861eed7006 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:57.570384 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 661/1342] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:23:57 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 267 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:23:57.581432 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce295a06-85c7-422b-87f1-47910e0e7f20 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:57.584666 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-ce295a06-85c7-422b-87f1-47910e0e7f20 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:57.714079 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ba311f-60e8-4103-b1c8-60861eed7006 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:57.715620 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-58ba311f-60e8-4103-b1c8-60861eed7006 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:57.799788 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce295a06-85c7-422b-87f1-47910e0e7f20 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:57.815235 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca54d80-dee7-45fd-93ee-79df46251878 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:57.816026 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca54d80-dee7-45fd-93ee-79df46251878 service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:57.817564 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca54d80-dee7-45fd-93ee-79df46251878 service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:57.818080 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca54d80-dee7-45fd-93ee-79df46251878 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:57.842190 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca54d80-dee7-45fd-93ee-79df46251878 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:57.844106 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca54d80-dee7-45fd-93ee-79df46251878 service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:57.879487 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 682/1343] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:23:57 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:23:57.898771 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ba311f-60e8-4103-b1c8-60861eed7006 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:57.905042 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b18775c-bcca-42be-9d6d-0e3573ff2a60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:57.905753 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b18775c-bcca-42be-9d6d-0e3573ff2a60 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:57.906163 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b18775c-bcca-42be-9d6d-0e3573ff2a60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:57.906630 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b18775c-bcca-42be-9d6d-0e3573ff2a60 admin admin] PATH_INFO: `/v3/projects/53adc5168fae411d8b4e6d74693d4f92` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:57.912522 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca54d80-dee7-45fd-93ee-79df46251878 service cinder] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:57.918129 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b18775c-bcca-42be-9d6d-0e3573ff2a60 admin admin] RBAC: Authorizing `identity:delete_project(project_id=53adc5168fae411d8b4e6d74693d4f92)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:57.929558 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca54d80-dee7-45fd-93ee-79df46251878 service cinder] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:57.929965 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b18775c-bcca-42be-9d6d-0e3573ff2a60 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:57.950184 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4867c4-45e6-4196-8297-d3a47ba54bbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:57.950913 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4867c4-45e6-4196-8297-d3a47ba54bbd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:57.951261 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4867c4-45e6-4196-8297-d3a47ba54bbd admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:57.951604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4867c4-45e6-4196-8297-d3a47ba54bbd admin admin] PATH_INFO: `/v3/users/5dad8decd91342cca9d4672f453dfff2` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:57.955332 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca4867c4-45e6-4196-8297-d3a47ba54bbd admin admin] RBAC: Authorizing `identity:delete_user(user_id=5dad8decd91342cca9d4672f453dfff2)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:57.964964 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca4867c4-45e6-4196-8297-d3a47ba54bbd admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:58.031511 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=96,keystone:INSERT=3,keystone:DELETE=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:23:58.089890 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1b18775c-bcca-42be-9d6d-0e3573ff2a60 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '53adc5168fae411d8b4e6d74693d4f92', 'request_id': 'req-1b18775c-bcca-42be-9d6d-0e3573ff2a60'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:58.102617 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1b18775c-bcca-42be-9d6d-0e3573ff2a60 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '53adc5168fae411d8b4e6d74693d4f92', 'request_id': 'req-1b18775c-bcca-42be-9d6d-0e3573ff2a60'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:58.111615 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1b18775c-bcca-42be-9d6d-0e3573ff2a60 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '53adc5168fae411d8b4e6d74693d4f92', 'request_id': 'req-1b18775c-bcca-42be-9d6d-0e3573ff2a60'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:58.130591 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-ca4867c4-45e6-4196-8297-d3a47ba54bbd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5dad8decd91342cca9d4672f453dfff2'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:58.135608 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1b18775c-bcca-42be-9d6d-0e3573ff2a60 admin admin] The token cache is being invalidate because project 53adc5168fae411d8b4e6d74693d4f92 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:23:58.135937 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1b18775c-bcca-42be-9d6d-0e3573ff2a60 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:58.139210 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1b18775c-bcca-42be-9d6d-0e3573ff2a60 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:58.142673 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 662/1344] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:23:57 2022] DELETE /identity/v3/projects/53adc5168fae411d8b4e6d74693d4f92 => generated 0 bytes in 429 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:23:58.148496 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b18775c-bcca-42be-9d6d-0e3573ff2a60 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:58.151538 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-1b18775c-bcca-42be-9d6d-0e3573ff2a60 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:58.163374 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-ca4867c4-45e6-4196-8297-d3a47ba54bbd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5dad8decd91342cca9d4672f453dfff2'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:58.190398 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-ca4867c4-45e6-4196-8297-d3a47ba54bbd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5dad8decd91342cca9d4672f453dfff2'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:58.191975 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-ca4867c4-45e6-4196-8297-d3a47ba54bbd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5dad8decd91342cca9d4672f453dfff2'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:58.198069 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-ca4867c4-45e6-4196-8297-d3a47ba54bbd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5dad8decd91342cca9d4672f453dfff2'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:58.231634 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 683/1345] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:23:57 2022] DELETE /identity/v3/users/5dad8decd91342cca9d4672f453dfff2 => generated 0 bytes in 320 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:23:58.248358 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c53012-e568-4588-9d85-31ab050c0bb2 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:58.248628 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c53012-e568-4588-9d85-31ab050c0bb2 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:58.248905 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c53012-e568-4588-9d85-31ab050c0bb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:58.294234 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-15c53012-e568-4588-9d85-31ab050c0bb2 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:23:58.295476 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-15c53012-e568-4588-9d85-31ab050c0bb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:58.353708 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b18775c-bcca-42be-9d6d-0e3573ff2a60 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:58.376035 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:23:58.451412 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 684/1346] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:23:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:23:58.503935 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9186f4-1be2-4d2a-993a-05d6b5ab5ba6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:58.505501 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9186f4-1be2-4d2a-993a-05d6b5ab5ba6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:58.505779 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9186f4-1be2-4d2a-993a-05d6b5ab5ba6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:58.506101 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9186f4-1be2-4d2a-993a-05d6b5ab5ba6 admin admin] PATH_INFO: `/v3/users/f9049e2900a94a489498b2105f45c8ef` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:58.507466 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9186f4-1be2-4d2a-993a-05d6b5ab5ba6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f9049e2900a94a489498b2105f45c8ef)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:58.509712 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9186f4-1be2-4d2a-993a-05d6b5ab5ba6 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:58.533655 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c53012-e568-4588-9d85-31ab050c0bb2 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:23:58.540268 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-15c53012-e568-4588-9d85-31ab050c0bb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:58.713375 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1d9186f4-1be2-4d2a-993a-05d6b5ab5ba6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f9049e2900a94a489498b2105f45c8ef'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:58.743688 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1d9186f4-1be2-4d2a-993a-05d6b5ab5ba6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f9049e2900a94a489498b2105f45c8ef'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:58.745136 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1d9186f4-1be2-4d2a-993a-05d6b5ab5ba6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f9049e2900a94a489498b2105f45c8ef'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:58.752791 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:23:58.776157 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1d9186f4-1be2-4d2a-993a-05d6b5ab5ba6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f9049e2900a94a489498b2105f45c8ef'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:58.793844 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c53012-e568-4588-9d85-31ab050c0bb2 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:23:58.801826 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a56b69-4beb-4c38-9e0f-bd0acb4a594d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:23:58.801826 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a56b69-4beb-4c38-9e0f-bd0acb4a594d service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:23:58.801826 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a56b69-4beb-4c38-9e0f-bd0acb4a594d service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:23:58.801826 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a56b69-4beb-4c38-9e0f-bd0acb4a594d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:23:58.803012 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-45a56b69-4beb-4c38-9e0f-bd0acb4a594d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:23:58.812755 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1d9186f4-1be2-4d2a-993a-05d6b5ab5ba6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f9049e2900a94a489498b2105f45c8ef'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:23:58.826316 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a56b69-4beb-4c38-9e0f-bd0acb4a594d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:23:58.830563 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 663/1347] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:23:58 2022] DELETE /identity/v3/users/f9049e2900a94a489498b2105f45c8ef => generated 0 bytes in 684 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:23:58.831389 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a56b69-4beb-4c38-9e0f-bd0acb4a594d service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:23:58.848561 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 685/1348] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:23:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:24:00.372535 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9186f4-1be2-4d2a-993a-05d6b5ab5ba6 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:00.374468 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-1d9186f4-1be2-4d2a-993a-05d6b5ab5ba6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:00.391274 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a56b69-4beb-4c38-9e0f-bd0acb4a594d service neutron] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:00.393080 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-45a56b69-4beb-4c38-9e0f-bd0acb4a594d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:00.439022 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a56b69-4beb-4c38-9e0f-bd0acb4a594d service neutron] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:00.609328 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9186f4-1be2-4d2a-993a-05d6b5ab5ba6 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:00.619223 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9122f31-0b88-45be-b420-bcdf22a9a33c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:00.620989 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9122f31-0b88-45be-b420-bcdf22a9a33c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:00.625804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9122f31-0b88-45be-b420-bcdf22a9a33c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:00.626178 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9122f31-0b88-45be-b420-bcdf22a9a33c admin admin] PATH_INFO: `/v3/projects/3fe71b3133b748c5b7b2c6442cc98a58` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:00.627682 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9122f31-0b88-45be-b420-bcdf22a9a33c admin admin] RBAC: Authorizing `identity:delete_project(project_id=3fe71b3133b748c5b7b2c6442cc98a58)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:00.632014 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f618a8f-38cc-4daa-b707-4e37070f66a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:00.632685 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f618a8f-38cc-4daa-b707-4e37070f66a5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:00.632971 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f618a8f-38cc-4daa-b707-4e37070f66a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:00.633272 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f618a8f-38cc-4daa-b707-4e37070f66a5 admin admin] PATH_INFO: `/v3/projects/2d929d321b214af7977d4bbc558b86d1` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:00.634651 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9122f31-0b88-45be-b420-bcdf22a9a33c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:00.637050 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f618a8f-38cc-4daa-b707-4e37070f66a5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2d929d321b214af7977d4bbc558b86d1)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:00.639029 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f618a8f-38cc-4daa-b707-4e37070f66a5 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:00.744263 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d9122f31-0b88-45be-b420-bcdf22a9a33c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3fe71b3133b748c5b7b2c6442cc98a58', 'request_id': 'req-d9122f31-0b88-45be-b420-bcdf22a9a33c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:00.756296 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-6f618a8f-38cc-4daa-b707-4e37070f66a5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2d929d321b214af7977d4bbc558b86d1', 'request_id': 'req-6f618a8f-38cc-4daa-b707-4e37070f66a5'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:00.771842 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d9122f31-0b88-45be-b420-bcdf22a9a33c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3fe71b3133b748c5b7b2c6442cc98a58', 'request_id': 'req-d9122f31-0b88-45be-b420-bcdf22a9a33c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:00.779364 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d9122f31-0b88-45be-b420-bcdf22a9a33c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3fe71b3133b748c5b7b2c6442cc98a58', 'request_id': 'req-d9122f31-0b88-45be-b420-bcdf22a9a33c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:00.781659 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-6f618a8f-38cc-4daa-b707-4e37070f66a5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2d929d321b214af7977d4bbc558b86d1', 'request_id': 'req-6f618a8f-38cc-4daa-b707-4e37070f66a5'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:00.792036 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-6f618a8f-38cc-4daa-b707-4e37070f66a5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2d929d321b214af7977d4bbc558b86d1', 'request_id': 'req-6f618a8f-38cc-4daa-b707-4e37070f66a5'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:00.803866 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d9122f31-0b88-45be-b420-bcdf22a9a33c admin admin] The token cache is being invalidate because project 3fe71b3133b748c5b7b2c6442cc98a58 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:24:00.804197 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d9122f31-0b88-45be-b420-bcdf22a9a33c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:00.804846 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d9122f31-0b88-45be-b420-bcdf22a9a33c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:00.811861 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 664/1349] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:24:00 2022] DELETE /identity/v3/projects/3fe71b3133b748c5b7b2c6442cc98a58 => generated 0 bytes in 438 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:00.817477 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bff5a94-963a-465e-a10e-16c2684a7fe6 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:00.817779 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bff5a94-963a-465e-a10e-16c2684a7fe6 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:00.818060 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bff5a94-963a-465e-a10e-16c2684a7fe6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:00.826897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-6f618a8f-38cc-4daa-b707-4e37070f66a5 admin admin] The token cache is being invalidate because project 2d929d321b214af7977d4bbc558b86d1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:24:00.827257 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-6f618a8f-38cc-4daa-b707-4e37070f66a5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:00.829908 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-6f618a8f-38cc-4daa-b707-4e37070f66a5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:00.831506 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 686/1350] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:24:00 2022] DELETE /identity/v3/projects/2d929d321b214af7977d4bbc558b86d1 => generated 0 bytes in 443 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:00.837095 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e08a098-b6fc-4ccb-9fad-676be5a4a7aa None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:00.837577 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e08a098-b6fc-4ccb-9fad-676be5a4a7aa None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:00.837945 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e08a098-b6fc-4ccb-9fad-676be5a4a7aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:00.886842 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-4bff5a94-963a-465e-a10e-16c2684a7fe6 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:00.892563 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-4bff5a94-963a-465e-a10e-16c2684a7fe6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:00.893758 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-5e08a098-b6fc-4ccb-9fad-676be5a4a7aa None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:00.897005 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-5e08a098-b6fc-4ccb-9fad-676be5a4a7aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:00.920139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:01.043199 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 687/1351] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:24:01.052296 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecae2f93-277b-4818-9363-c2fb12070243 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:01.052585 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecae2f93-277b-4818-9363-c2fb12070243 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:01.052861 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecae2f93-277b-4818-9363-c2fb12070243 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:01.063083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 665/1352] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 249 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:01.071224 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88137118-0428-454b-ac2f-917c3964f3e3 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:01.071507 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88137118-0428-454b-ac2f-917c3964f3e3 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:01.071798 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88137118-0428-454b-ac2f-917c3964f3e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:01.117222 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-ecae2f93-277b-4818-9363-c2fb12070243 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:01.118429 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-ecae2f93-277b-4818-9363-c2fb12070243 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:01.126786 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-88137118-0428-454b-ac2f-917c3964f3e3 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:01.131326 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-88137118-0428-454b-ac2f-917c3964f3e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:01.137838 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 688/1353] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:01.141028 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecae2f93-277b-4818-9363-c2fb12070243 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:01.154755 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 666/1354] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:24:01.158804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88137118-0428-454b-ac2f-917c3964f3e3 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:01.168312 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecae2f93-277b-4818-9363-c2fb12070243 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:01.174415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:01.176917 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408345b7-8984-4691-993c-052f77c1fe30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:01.178653 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408345b7-8984-4691-993c-052f77c1fe30 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:01.178653 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408345b7-8984-4691-993c-052f77c1fe30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:01.178653 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408345b7-8984-4691-993c-052f77c1fe30 admin admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:01.179408 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408345b7-8984-4691-993c-052f77c1fe30 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:24:01.179651 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408345b7-8984-4691-993c-052f77c1fe30 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:01.181559 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408345b7-8984-4691-993c-052f77c1fe30 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:01.189198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88137118-0428-454b-ac2f-917c3964f3e3 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:01.202758 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c66725d4-6350-4432-a5b3-0eedb3585c70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:01.203565 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c66725d4-6350-4432-a5b3-0eedb3585c70 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:01.203854 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c66725d4-6350-4432-a5b3-0eedb3585c70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:01.207699 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c66725d4-6350-4432-a5b3-0eedb3585c70 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:01.208433 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c66725d4-6350-4432-a5b3-0eedb3585c70 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:24:01.208746 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c66725d4-6350-4432-a5b3-0eedb3585c70 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:01.210471 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c66725d4-6350-4432-a5b3-0eedb3585c70 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:01.218913 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 689/1355] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:24:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:24:01.219848 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408345b7-8984-4691-993c-052f77c1fe30 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:01.244063 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 667/1356] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:24:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:24:01.248442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c66725d4-6350-4432-a5b3-0eedb3585c70 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:01.249405 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408345b7-8984-4691-993c-052f77c1fe30 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:01.266367 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5d237b-dbaf-4c9f-83b4-877b293d58a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:01.266653 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c66725d4-6350-4432-a5b3-0eedb3585c70 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:01.266912 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5d237b-dbaf-4c9f-83b4-877b293d58a5 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:01.267194 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5d237b-dbaf-4c9f-83b4-877b293d58a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:01.267465 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5d237b-dbaf-4c9f-83b4-877b293d58a5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:01.268123 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc5d237b-dbaf-4c9f-83b4-877b293d58a5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:01.272512 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc5d237b-dbaf-4c9f-83b4-877b293d58a5 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:01.278481 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f57ae5-3522-47b0-a5e8-7592328920e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:01.279485 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f57ae5-3522-47b0-a5e8-7592328920e2 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:01.279778 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f57ae5-3522-47b0-a5e8-7592328920e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:01.281306 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f57ae5-3522-47b0-a5e8-7592328920e2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:01.282490 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f57ae5-3522-47b0-a5e8-7592328920e2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:01.287834 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f57ae5-3522-47b0-a5e8-7592328920e2 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:01.334442 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 690/1357] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:24:01 2022] POST /identity/v3/projects => generated 383 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:24:01.340138 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5d237b-dbaf-4c9f-83b4-877b293d58a5 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:01.358710 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 668/1358] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:24:01 2022] POST /identity/v3/projects => generated 379 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:24:01.362580 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f57ae5-3522-47b0-a5e8-7592328920e2 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:01.500967 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5d237b-dbaf-4c9f-83b4-877b293d58a5 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:01.540892 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f57ae5-3522-47b0-a5e8-7592328920e2 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:01.547229 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b5b86a8-ce45-4764-9c3d-ba71e232ed97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:01.548691 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5b86a8-ce45-4764-9c3d-ba71e232ed97 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:01.549228 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5b86a8-ce45-4764-9c3d-ba71e232ed97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:01.549669 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5b86a8-ce45-4764-9c3d-ba71e232ed97 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:01.553160 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b5b86a8-ce45-4764-9c3d-ba71e232ed97 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:01.556005 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b5b86a8-ce45-4764-9c3d-ba71e232ed97 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:01.619529 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 669/1359] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:24:01 2022] POST /identity/v3/users => generated 290 bytes in 257 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:24:01.621005 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b5b86a8-ce45-4764-9c3d-ba71e232ed97 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:01.671199 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b5b86a8-ce45-4764-9c3d-ba71e232ed97 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:01.691974 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2441dee-4d9d-4f48-8e37-e0998096b241 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:01.695400 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2441dee-4d9d-4f48-8e37-e0998096b241 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:01.697095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2441dee-4d9d-4f48-8e37-e0998096b241 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:01.698447 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2441dee-4d9d-4f48-8e37-e0998096b241 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:01.702572 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2441dee-4d9d-4f48-8e37-e0998096b241 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:01.708095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2441dee-4d9d-4f48-8e37-e0998096b241 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:01.746929 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 670/1360] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:01 2022] GET /identity/v3/roles => generated 1420 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:01.748808 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2441dee-4d9d-4f48-8e37-e0998096b241 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:01.766378 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b2f5e8-26e4-4e9e-9c74-0e2449d3fc22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:01.771119 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b2f5e8-26e4-4e9e-9c74-0e2449d3fc22 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:01.771720 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b2f5e8-26e4-4e9e-9c74-0e2449d3fc22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:01.772220 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b2f5e8-26e4-4e9e-9c74-0e2449d3fc22 admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:01.774403 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2441dee-4d9d-4f48-8e37-e0998096b241 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:01.774705 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b2f5e8-26e4-4e9e-9c74-0e2449d3fc22 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:01.779863 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b2f5e8-26e4-4e9e-9c74-0e2449d3fc22 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:01.780510 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b36cf6-7976-49ba-bf70-5ac5c4990885 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:01.781447 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b36cf6-7976-49ba-bf70-5ac5c4990885 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:01.781881 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b36cf6-7976-49ba-bf70-5ac5c4990885 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:01.781881 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b36cf6-7976-49ba-bf70-5ac5c4990885 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:01.782623 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b36cf6-7976-49ba-bf70-5ac5c4990885 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:01.784270 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b36cf6-7976-49ba-bf70-5ac5c4990885 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:01.828491 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 671/1361] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:01 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:01.833981 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b36cf6-7976-49ba-bf70-5ac5c4990885 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:01.842084 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 691/1362] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:24:01 2022] POST /identity/v3/users => generated 292 bytes in 502 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:24:01.843661 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b2f5e8-26e4-4e9e-9c74-0e2449d3fc22 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:01.856134 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b2f5e8-26e4-4e9e-9c74-0e2449d3fc22 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:01.862741 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b36cf6-7976-49ba-bf70-5ac5c4990885 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:01.876839 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71848f20-e945-47e2-a9b5-1995976e8448 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:01.883749 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410f1856-f99b-446e-8963-e7b1f7e55d26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:01.884590 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410f1856-f99b-446e-8963-e7b1f7e55d26 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:01.884956 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410f1856-f99b-446e-8963-e7b1f7e55d26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:01.885320 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410f1856-f99b-446e-8963-e7b1f7e55d26 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:01.886218 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410f1856-f99b-446e-8963-e7b1f7e55d26 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:01.887984 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410f1856-f99b-446e-8963-e7b1f7e55d26 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:01.897639 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:01.898774 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71848f20-e945-47e2-a9b5-1995976e8448 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:01.899152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71848f20-e945-47e2-a9b5-1995976e8448 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:01.900784 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71848f20-e945-47e2-a9b5-1995976e8448 admin admin] PATH_INFO: `/v3/projects/21e86216310241f987e86f2eab46b3e4/users/1023c639b6bc4f448c30c7cee2765788/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:01.942083 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 692/1363] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:01 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:01.944125 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410f1856-f99b-446e-8963-e7b1f7e55d26 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:01.953329 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71848f20-e945-47e2-a9b5-1995976e8448 admin admin] RBAC: Authorizing `identity:create_grant(project_id=21e86216310241f987e86f2eab46b3e4, user_id=1023c639b6bc4f448c30c7cee2765788, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:01.961667 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71848f20-e945-47e2-a9b5-1995976e8448 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:01.979663 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410f1856-f99b-446e-8963-e7b1f7e55d26 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:02.003195 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 672/1364] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:24:01 2022] PUT /identity/v3/projects/21e86216310241f987e86f2eab46b3e4/users/1023c639b6bc4f448c30c7cee2765788/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:02.015533 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c5e6be-2b0b-4e27-983c-eba5c921e296 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:02.016172 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c5e6be-2b0b-4e27-983c-eba5c921e296 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:02.016680 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c5e6be-2b0b-4e27-983c-eba5c921e296 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:02.094996 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-17c5e6be-2b0b-4e27-983c-eba5c921e296 None None] MFA Rules not processed for user `1023c639b6bc4f448c30c7cee2765788`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:02.099732 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-17c5e6be-2b0b-4e27-983c-eba5c921e296 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:02.130923 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-529dd56b-ea3c-4926-a6fe-861d90063f3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:02.131556 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529dd56b-ea3c-4926-a6fe-861d90063f3a admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:02.131836 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529dd56b-ea3c-4926-a6fe-861d90063f3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:02.132110 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529dd56b-ea3c-4926-a6fe-861d90063f3a admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:02.132830 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-529dd56b-ea3c-4926-a6fe-861d90063f3a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:02.135570 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-529dd56b-ea3c-4926-a6fe-861d90063f3a admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:02.158576 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 693/1365] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:01 2022] GET /identity/v3/roles => generated 1420 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:02.172833 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-529dd56b-ea3c-4926-a6fe-861d90063f3a admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:02.210257 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-529dd56b-ea3c-4926-a6fe-861d90063f3a admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:02.226829 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10bc434-cd08-4ea6-8e0c-3bc18be511c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:02.227676 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10bc434-cd08-4ea6-8e0c-3bc18be511c2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:02.228254 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10bc434-cd08-4ea6-8e0c-3bc18be511c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:02.228533 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10bc434-cd08-4ea6-8e0c-3bc18be511c2 admin admin] PATH_INFO: `/v3/projects/9e496b7490434d1cb34d22fe39c26e54/users/7bbfb93f65f64b9b9fd686cf13776c63/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:02.257500 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10bc434-cd08-4ea6-8e0c-3bc18be511c2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9e496b7490434d1cb34d22fe39c26e54, user_id=7bbfb93f65f64b9b9fd686cf13776c63, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:02.261312 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10bc434-cd08-4ea6-8e0c-3bc18be511c2 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:02.279295 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:02.294706 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 673/1366] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:02 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 282 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:02.298818 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c5e6be-2b0b-4e27-983c-eba5c921e296 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:02.311884 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 694/1367] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:24:02 2022] PUT /identity/v3/projects/9e496b7490434d1cb34d22fe39c26e54/users/7bbfb93f65f64b9b9fd686cf13776c63/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:02.317624 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c5e6be-2b0b-4e27-983c-eba5c921e296 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:02.326874 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e60e7e-054b-45d6-8a69-c7a3e366fbf9 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:02.327180 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e60e7e-054b-45d6-8a69-c7a3e366fbf9 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:02.327435 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e60e7e-054b-45d6-8a69-c7a3e366fbf9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:02.414168 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-57e60e7e-054b-45d6-8a69-c7a3e366fbf9 None None] MFA Rules not processed for user `7bbfb93f65f64b9b9fd686cf13776c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:02.415427 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-57e60e7e-054b-45d6-8a69-c7a3e366fbf9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:02.514363 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab285e8-1edb-4f50-a5e9-bbb8e557f646 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:02.517225 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab285e8-1edb-4f50-a5e9-bbb8e557f646 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:02.517756 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab285e8-1edb-4f50-a5e9-bbb8e557f646 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:02.519648 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab285e8-1edb-4f50-a5e9-bbb8e557f646 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:02.521479 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab285e8-1edb-4f50-a5e9-bbb8e557f646 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:02.528707 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab285e8-1edb-4f50-a5e9-bbb8e557f646 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:02.606283 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 674/1368] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:24:02 2022] POST /identity/v3/projects => generated 377 bytes in 308 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:24:02.609660 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab285e8-1edb-4f50-a5e9-bbb8e557f646 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:02.714967 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 695/1369] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:02 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 386 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:02.729336 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e60e7e-054b-45d6-8a69-c7a3e366fbf9 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:02.739674 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-57e60e7e-054b-45d6-8a69-c7a3e366fbf9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:02.826236 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab285e8-1edb-4f50-a5e9-bbb8e557f646 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:02.845917 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c9f89d-22c1-47a9-a4a1-87e2fe755973 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:02.846555 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c9f89d-22c1-47a9-a4a1-87e2fe755973 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:02.846836 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c9f89d-22c1-47a9-a4a1-87e2fe755973 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:02.847109 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c9f89d-22c1-47a9-a4a1-87e2fe755973 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:02.847840 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c9f89d-22c1-47a9-a4a1-87e2fe755973 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:02.849845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c9f89d-22c1-47a9-a4a1-87e2fe755973 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:02.925522 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 675/1370] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:24:02 2022] POST /identity/v3/users => generated 289 bytes in 315 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:24:02.928809 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c9f89d-22c1-47a9-a4a1-87e2fe755973 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:02.951920 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c9f89d-22c1-47a9-a4a1-87e2fe755973 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:02.959411 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47234ea-61d5-41c1-aa3f-5b8fde4e9aa8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:02.960051 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47234ea-61d5-41c1-aa3f-5b8fde4e9aa8 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:02.960331 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47234ea-61d5-41c1-aa3f-5b8fde4e9aa8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:02.960607 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47234ea-61d5-41c1-aa3f-5b8fde4e9aa8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:02.961314 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e47234ea-61d5-41c1-aa3f-5b8fde4e9aa8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:02.963252 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e47234ea-61d5-41c1-aa3f-5b8fde4e9aa8 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:02.964928 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e60e7e-054b-45d6-8a69-c7a3e366fbf9 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:02.999260 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:03.011850 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 676/1371] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:02 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:03.014122 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47234ea-61d5-41c1-aa3f-5b8fde4e9aa8 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:03.047392 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47234ea-61d5-41c1-aa3f-5b8fde4e9aa8 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:03.066074 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d85ac33-95c8-486f-971b-c32d3e83f744 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:03.067017 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d85ac33-95c8-486f-971b-c32d3e83f744 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:03.067300 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d85ac33-95c8-486f-971b-c32d3e83f744 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:03.067573 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d85ac33-95c8-486f-971b-c32d3e83f744 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:03.075065 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d85ac33-95c8-486f-971b-c32d3e83f744 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:03.076748 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d85ac33-95c8-486f-971b-c32d3e83f744 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:03.127550 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 677/1372] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:03 2022] GET /identity/v3/roles => generated 1420 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:03.130676 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d85ac33-95c8-486f-971b-c32d3e83f744 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:03.134040 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217da5c2-319e-417c-ad3d-a916fdadd55a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:03.134687 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217da5c2-319e-417c-ad3d-a916fdadd55a service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:03.134969 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217da5c2-319e-417c-ad3d-a916fdadd55a service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:03.137214 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217da5c2-319e-417c-ad3d-a916fdadd55a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:03.157517 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d85ac33-95c8-486f-971b-c32d3e83f744 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:03.174658 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c09875-37af-4377-abb1-2f4d22e0bcd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:03.175413 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c09875-37af-4377-abb1-2f4d22e0bcd9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:03.176196 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c09875-37af-4377-abb1-2f4d22e0bcd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:03.177096 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c09875-37af-4377-abb1-2f4d22e0bcd9 admin admin] PATH_INFO: `/v3/projects/5036cbc5dab947429bb10d03b1abd17e/users/53e3e7704920474bbda4485d33337ef7/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:03.223146 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c09875-37af-4377-abb1-2f4d22e0bcd9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5036cbc5dab947429bb10d03b1abd17e, user_id=53e3e7704920474bbda4485d33337ef7, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:03.226725 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c09875-37af-4377-abb1-2f4d22e0bcd9 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:03.258573 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 678/1373] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:24:03 2022] PUT /identity/v3/projects/5036cbc5dab947429bb10d03b1abd17e/users/53e3e7704920474bbda4485d33337ef7/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:03.271584 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e842277-9e41-4f7f-be2e-b2e732dabb23 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:03.272200 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e842277-9e41-4f7f-be2e-b2e732dabb23 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:03.272810 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e842277-9e41-4f7f-be2e-b2e732dabb23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:03.336649 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217da5c2-319e-417c-ad3d-a916fdadd55a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:03.338083 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217da5c2-319e-417c-ad3d-a916fdadd55a service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:03.346389 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-4e842277-9e41-4f7f-be2e-b2e732dabb23 None None] MFA Rules not processed for user `53e3e7704920474bbda4485d33337ef7`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:03.351573 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-4e842277-9e41-4f7f-be2e-b2e732dabb23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:03.372152 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 696/1374] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:02 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 643 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:24:03.406939 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:03.635043 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 679/1375] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:03 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 366 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:03.640115 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217da5c2-319e-417c-ad3d-a916fdadd55a service cinder] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:03.664438 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217da5c2-319e-417c-ad3d-a916fdadd55a service cinder] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:03.778145 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803122b5-6749-496f-8d47-bab61059334b None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:03.778437 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803122b5-6749-496f-8d47-bab61059334b None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:03.778716 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803122b5-6749-496f-8d47-bab61059334b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:03.858387 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b224da60-abf2-431e-a5f4-e1200de307b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:03.859867 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b224da60-abf2-431e-a5f4-e1200de307b4 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:03.860445 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b224da60-abf2-431e-a5f4-e1200de307b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:03.860890 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b224da60-abf2-431e-a5f4-e1200de307b4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:03.861818 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b224da60-abf2-431e-a5f4-e1200de307b4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:03.870376 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b224da60-abf2-431e-a5f4-e1200de307b4 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:03.892747 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-803122b5-6749-496f-8d47-bab61059334b None None] MFA Rules not processed for user `68909cf0c5bf4875927b30d9a6ce8cd9`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:03.910135 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-803122b5-6749-496f-8d47-bab61059334b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:03.958971 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 697/1376] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:24:03 2022] POST /identity/v3/projects => generated 377 bytes in 316 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:24:03.962635 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b224da60-abf2-431e-a5f4-e1200de307b4 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:04.151118 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:04.167923 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b224da60-abf2-431e-a5f4-e1200de307b4 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:04.174301 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 680/1377] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:03 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 397 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:24:04.194018 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ffe0b66-a08a-487e-81ce-ce03bdd6e198 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:04.194383 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ffe0b66-a08a-487e-81ce-ce03bdd6e198 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:04.194728 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ffe0b66-a08a-487e-81ce-ce03bdd6e198 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:04.196562 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85993adf-81cd-4019-8655-9971a53be784 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:04.197218 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85993adf-81cd-4019-8655-9971a53be784 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:04.197495 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85993adf-81cd-4019-8655-9971a53be784 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:04.197762 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85993adf-81cd-4019-8655-9971a53be784 admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:04.198406 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85993adf-81cd-4019-8655-9971a53be784 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:04.203983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85993adf-81cd-4019-8655-9971a53be784 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:04.253087 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 698/1378] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:24:03 2022] POST /identity/v3/users => generated 295 bytes in 293 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:24:04.257795 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85993adf-81cd-4019-8655-9971a53be784 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:04.272007 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-4ffe0b66-a08a-487e-81ce-ce03bdd6e198 None None] MFA Rules not processed for user `68909cf0c5bf4875927b30d9a6ce8cd9`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:04.276585 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85993adf-81cd-4019-8655-9971a53be784 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:04.280271 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-4ffe0b66-a08a-487e-81ce-ce03bdd6e198 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:04.286806 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556586ee-4b38-4b86-b896-6d246a16fd03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:04.287537 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556586ee-4b38-4b86-b896-6d246a16fd03 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:04.287832 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556586ee-4b38-4b86-b896-6d246a16fd03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:04.288091 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556586ee-4b38-4b86-b896-6d246a16fd03 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:04.288737 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-556586ee-4b38-4b86-b896-6d246a16fd03 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:04.290491 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-556586ee-4b38-4b86-b896-6d246a16fd03 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:04.326106 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 699/1379] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:04 2022] GET /identity/v3/roles => generated 1420 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:04.329254 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556586ee-4b38-4b86-b896-6d246a16fd03 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:04.356370 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556586ee-4b38-4b86-b896-6d246a16fd03 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:04.368564 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193ff6f4-a1bc-4360-953f-43db48e98e2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:04.369248 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193ff6f4-a1bc-4360-953f-43db48e98e2d admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:04.369529 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193ff6f4-a1bc-4360-953f-43db48e98e2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:04.369802 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193ff6f4-a1bc-4360-953f-43db48e98e2d admin admin] PATH_INFO: `/v3/projects/b2122c27e4ce4f149fed26e36cc0c360/users/048798141e664df8835afdd614a5d413/roles/414286927fdd44688107ff5e10235acf` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:04.409956 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-193ff6f4-a1bc-4360-953f-43db48e98e2d admin admin] RBAC: Authorizing `identity:create_grant(project_id=b2122c27e4ce4f149fed26e36cc0c360, user_id=048798141e664df8835afdd614a5d413, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:04.416544 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-193ff6f4-a1bc-4360-953f-43db48e98e2d admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:04.436953 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 681/1380] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:04 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 244 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:24:04.448539 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 700/1381] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:24:04 2022] PUT /identity/v3/projects/b2122c27e4ce4f149fed26e36cc0c360/users/048798141e664df8835afdd614a5d413/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:04.452229 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ffe0b66-a08a-487e-81ce-ce03bdd6e198 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:04.456483 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193ff6f4-a1bc-4360-953f-43db48e98e2d admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:04.459092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-193ff6f4-a1bc-4360-953f-43db48e98e2d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:04.479556 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ffe0b66-a08a-487e-81ce-ce03bdd6e198 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:04.605881 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:04.631406 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54acd67c-db6e-42e8-98dc-a215138fdc65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:04.632049 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54acd67c-db6e-42e8-98dc-a215138fdc65 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:04.632328 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54acd67c-db6e-42e8-98dc-a215138fdc65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:04.632596 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54acd67c-db6e-42e8-98dc-a215138fdc65 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:04.633392 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54acd67c-db6e-42e8-98dc-a215138fdc65 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:04.635194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54acd67c-db6e-42e8-98dc-a215138fdc65 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:04.663650 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 682/1382] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:04 2022] GET /identity/v3/roles => generated 1420 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:04.664988 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54acd67c-db6e-42e8-98dc-a215138fdc65 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:04.703211 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193ff6f4-a1bc-4360-953f-43db48e98e2d admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:04.717633 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54acd67c-db6e-42e8-98dc-a215138fdc65 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:04.719750 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8cab2ae-4212-44c9-9102-032019a60f84 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:04.721815 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8cab2ae-4212-44c9-9102-032019a60f84 service nova] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:04.722317 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8cab2ae-4212-44c9-9102-032019a60f84 service nova] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:04.723828 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8cab2ae-4212-44c9-9102-032019a60f84 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:04.743899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d870a643-cb38-42cb-8bef-2edaf2cfa25a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:04.744646 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d870a643-cb38-42cb-8bef-2edaf2cfa25a admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:04.744928 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d870a643-cb38-42cb-8bef-2edaf2cfa25a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:04.745240 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d870a643-cb38-42cb-8bef-2edaf2cfa25a admin admin] PATH_INFO: `/v3/projects/b2122c27e4ce4f149fed26e36cc0c360/users/048798141e664df8835afdd614a5d413/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:04.764340 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d870a643-cb38-42cb-8bef-2edaf2cfa25a admin admin] RBAC: Authorizing `identity:create_grant(project_id=b2122c27e4ce4f149fed26e36cc0c360, user_id=048798141e664df8835afdd614a5d413, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:04.769402 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d870a643-cb38-42cb-8bef-2edaf2cfa25a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:04.830710 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 683/1383] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:24:04 2022] PUT /identity/v3/projects/b2122c27e4ce4f149fed26e36cc0c360/users/048798141e664df8835afdd614a5d413/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:04.841221 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d2a947-8f72-45d6-bea2-2b2f1f0af70a None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:04.842000 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d2a947-8f72-45d6-bea2-2b2f1f0af70a None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:04.842450 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d2a947-8f72-45d6-bea2-2b2f1f0af70a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:04.951510 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-b6d2a947-8f72-45d6-bea2-2b2f1f0af70a None None] MFA Rules not processed for user `048798141e664df8835afdd614a5d413`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:04.957625 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-b6d2a947-8f72-45d6-bea2-2b2f1f0af70a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:04.974218 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8cab2ae-4212-44c9-9102-032019a60f84 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:04.979917 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8cab2ae-4212-44c9-9102-032019a60f84 service nova] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:04.999678 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 701/1384] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:04 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 546 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 27 01:24:05.121792 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 684/1385] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:04 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 281 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:24:05.140341 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8cab2ae-4212-44c9-9102-032019a60f84 service nova] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:05.281323 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:05.292143 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8cab2ae-4212-44c9-9102-032019a60f84 service nova] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:05.345711 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea06ee2-aa5c-4469-9d4e-50b661994b63 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:05.345996 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea06ee2-aa5c-4469-9d4e-50b661994b63 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:05.346593 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea06ee2-aa5c-4469-9d4e-50b661994b63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:05.416153 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd87358-6bc7-468b-9dc9-d3a12027ad23 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:05.416921 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd87358-6bc7-468b-9dc9-d3a12027ad23 service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:05.417255 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd87358-6bc7-468b-9dc9-d3a12027ad23 service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:05.417596 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd87358-6bc7-468b-9dc9-d3a12027ad23 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:05.425263 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-7ea06ee2-aa5c-4469-9d4e-50b661994b63 None None] MFA Rules not processed for user `99bcd22ff73b4526a54cf0e76f509436`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:05.427463 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-7ea06ee2-aa5c-4469-9d4e-50b661994b63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:05.453662 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd87358-6bc7-468b-9dc9-d3a12027ad23 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:05.453662 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd87358-6bc7-468b-9dc9-d3a12027ad23 service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:05.464389 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 702/1386] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:05 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:24:05.628905 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd87358-6bc7-468b-9dc9-d3a12027ad23 service cinder] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:05.631218 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-4fd87358-6bc7-468b-9dc9-d3a12027ad23 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:05.651399 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd87358-6bc7-468b-9dc9-d3a12027ad23 service cinder] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:05.663879 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0022e3df-fa5a-4cc5-aaf8-745b5703c418 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:05.668202 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0022e3df-fa5a-4cc5-aaf8-745b5703c418 service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:05.669474 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0022e3df-fa5a-4cc5-aaf8-745b5703c418 service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:05.670312 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0022e3df-fa5a-4cc5-aaf8-745b5703c418 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:05.713736 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 685/1387] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:05 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 369 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:05.765442 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0022e3df-fa5a-4cc5-aaf8-745b5703c418 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:05.767312 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0022e3df-fa5a-4cc5-aaf8-745b5703c418 service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:05.778877 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 703/1388] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:05 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 150 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:24:06.056723 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea06ee2-aa5c-4469-9d4e-50b661994b63 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:06.058189 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-7ea06ee2-aa5c-4469-9d4e-50b661994b63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:06.157625 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:06.172182 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b979d5-a3fb-4b38-a7b3-6913b29e1d2b None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:06.172747 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b979d5-a3fb-4b38-a7b3-6913b29e1d2b None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:06.172747 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b979d5-a3fb-4b38-a7b3-6913b29e1d2b None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:06.174062 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 704/1389] 10.222.0.42 () {54 vars in 912 bytes} [Thu Oct 27 01:24:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 27 01:24:06.187132 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e4c6cd-0f37-4494-bbd8-c47a53db9cc1 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:06.187480 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e4c6cd-0f37-4494-bbd8-c47a53db9cc1 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:06.187671 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e4c6cd-0f37-4494-bbd8-c47a53db9cc1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:06.216245 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea06ee2-aa5c-4469-9d4e-50b661994b63 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:06.233513 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4be59b7-a2d2-4061-b353-0fa894af0cf0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:06.234556 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4be59b7-a2d2-4061-b353-0fa894af0cf0 service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:06.235223 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4be59b7-a2d2-4061-b353-0fa894af0cf0 service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:06.236941 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4be59b7-a2d2-4061-b353-0fa894af0cf0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:06.254923 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-42e4c6cd-0f37-4494-bbd8-c47a53db9cc1 None None] MFA Rules not processed for user `4072417990c94b0488871332d28224ba`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:06.256191 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-42e4c6cd-0f37-4494-bbd8-c47a53db9cc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:06.262309 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4be59b7-a2d2-4061-b353-0fa894af0cf0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:06.266481 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4be59b7-a2d2-4061-b353-0fa894af0cf0 service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:06.292898 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 686/1390] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:06 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 237 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:24:06.326128 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4571b92b-b5fe-4eb7-bec9-26b4bac2812f None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:06.326418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4571b92b-b5fe-4eb7-bec9-26b4bac2812f None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:06.326672 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4571b92b-b5fe-4eb7-bec9-26b4bac2812f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:06.349480 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 705/1391] 10.222.0.42 () {58 vars in 1009 bytes} [Thu Oct 27 01:24:06 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:06.367336 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e4c6cd-0f37-4494-bbd8-c47a53db9cc1 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:06.371839 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-42e4c6cd-0f37-4494-bbd8-c47a53db9cc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:06.396775 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e4c6cd-0f37-4494-bbd8-c47a53db9cc1 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:06.400791 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-4571b92b-b5fe-4eb7-bec9-26b4bac2812f None None] MFA Rules not processed for user `d311fc1f82124819b509f766fa432e2c`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:06.403405 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-4571b92b-b5fe-4eb7-bec9-26b4bac2812f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:06.405772 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30aeae1-3e8e-4a1c-bdd8-d881c622e4d2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:06.408125 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30aeae1-3e8e-4a1c-bdd8-d881c622e4d2 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:06.408620 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30aeae1-3e8e-4a1c-bdd8-d881c622e4d2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:06.409517 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30aeae1-3e8e-4a1c-bdd8-d881c622e4d2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:06.434532 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30aeae1-3e8e-4a1c-bdd8-d881c622e4d2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:06.436065 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30aeae1-3e8e-4a1c-bdd8-d881c622e4d2 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:06.458374 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 706/1392] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:06 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:24:06.581094 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 687/1393] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:06 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 256 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:06.597971 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a3eada-44c7-44a6-8350-a80cfd01efe1 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:06.597971 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a3eada-44c7-44a6-8350-a80cfd01efe1 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:06.597971 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a3eada-44c7-44a6-8350-a80cfd01efe1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:06.642354 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-b0a3eada-44c7-44a6-8350-a80cfd01efe1 None None] MFA Rules not processed for user `d311fc1f82124819b509f766fa432e2c`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:06.643594 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-b0a3eada-44c7-44a6-8350-a80cfd01efe1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:06.666887 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 707/1394] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:06 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 72 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:06.871713 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4571b92b-b5fe-4eb7-bec9-26b4bac2812f None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:06.904852 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4571b92b-b5fe-4eb7-bec9-26b4bac2812f None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:06.929899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a29bb5-5744-4e10-a99c-109d4db6e2f1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:06.933608 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a29bb5-5744-4e10-a99c-109d4db6e2f1 service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:06.933962 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a29bb5-5744-4e10-a99c-109d4db6e2f1 service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:06.935101 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a29bb5-5744-4e10-a99c-109d4db6e2f1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:06.968910 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a29bb5-5744-4e10-a99c-109d4db6e2f1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:06.971223 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a29bb5-5744-4e10-a99c-109d4db6e2f1 service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:06.994265 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 688/1395] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:06 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:24:07.447321 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82b5159-84b4-48fa-a975-315b4ddf62bb None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:07.447808 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82b5159-84b4-48fa-a975-315b4ddf62bb None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:07.447905 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82b5159-84b4-48fa-a975-315b4ddf62bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:07.518766 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-d82b5159-84b4-48fa-a975-315b4ddf62bb None None] MFA Rules not processed for user `cca36a277e3643a498b27e3c5d5e14f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:07.523105 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-d82b5159-84b4-48fa-a975-315b4ddf62bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:07.548601 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:07.655408 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a29bb5-5744-4e10-a99c-109d4db6e2f1 service nova] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:07.657798 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-09a29bb5-5744-4e10-a99c-109d4db6e2f1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:07.768685 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 708/1396] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:07 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 322 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:07.777563 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab12dc87-8d22-4b10-ab6c-e116e46d5344 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:07.784738 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab12dc87-8d22-4b10-ab6c-e116e46d5344 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:07.786020 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab12dc87-8d22-4b10-ab6c-e116e46d5344 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:07.787583 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 709/1397] 10.222.0.42 () {54 vars in 912 bytes} [Thu Oct 27 01:24:07 2022] GET /identity => generated 270 bytes in 18 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:24:07.795186 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5831c176-aabc-4595-8cac-a17789a97140 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:07.795813 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5831c176-aabc-4595-8cac-a17789a97140 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:07.796340 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5831c176-aabc-4595-8cac-a17789a97140 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:07.865712 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-5831c176-aabc-4595-8cac-a17789a97140 None None] MFA Rules not processed for user `4072417990c94b0488871332d28224ba`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:07.866967 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-5831c176-aabc-4595-8cac-a17789a97140 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:07.919715 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 710/1398] 10.222.0.42 () {58 vars in 1009 bytes} [Thu Oct 27 01:24:07 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:07.932174 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5831c176-aabc-4595-8cac-a17789a97140 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:07.952582 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a29bb5-5744-4e10-a99c-109d4db6e2f1 service nova] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:07.955662 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5831c176-aabc-4595-8cac-a17789a97140 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:07.967366 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23cf6bd4-9389-4e39-ab27-8e6370870adb service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62051562fc0b4cd08750d2293e71787c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:07.967559 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b114ec-7698-4278-aac6-948754c561a0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:07.968358 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b114ec-7698-4278-aac6-948754c561a0 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:07.968712 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b114ec-7698-4278-aac6-948754c561a0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:07.969040 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b114ec-7698-4278-aac6-948754c561a0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:07.969264 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23cf6bd4-9389-4e39-ab27-8e6370870adb service placement] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:07.969605 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23cf6bd4-9389-4e39-ab27-8e6370870adb service placement] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:07.969883 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23cf6bd4-9389-4e39-ab27-8e6370870adb service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:07.975449 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-23cf6bd4-9389-4e39-ab27-8e6370870adb service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:07.985769 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b114ec-7698-4278-aac6-948754c561a0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:07.991146 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b114ec-7698-4278-aac6-948754c561a0 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:07.995946 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23cf6bd4-9389-4e39-ab27-8e6370870adb service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:08.001901 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23cf6bd4-9389-4e39-ab27-8e6370870adb service placement] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:08.034442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 689/1399] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:07 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 380 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 27 01:24:08.051259 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 711/1400] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:07 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 27 01:24:08.242195 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23cf6bd4-9389-4e39-ab27-8e6370870adb service placement] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:08.267777 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23cf6bd4-9389-4e39-ab27-8e6370870adb service placement] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:08.282752 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff606d62-d0d5-4801-a08e-83078ea94d69 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:08.284034 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff606d62-d0d5-4801-a08e-83078ea94d69 service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:08.285333 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff606d62-d0d5-4801-a08e-83078ea94d69 service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:08.285666 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff606d62-d0d5-4801-a08e-83078ea94d69 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:08.310169 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff606d62-d0d5-4801-a08e-83078ea94d69 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:08.317366 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff606d62-d0d5-4801-a08e-83078ea94d69 service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:08.357505 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 690/1401] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:08 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 27 01:24:08.892238 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b114ec-7698-4278-aac6-948754c561a0 service neutron] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:08.892238 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-b5b114ec-7698-4278-aac6-948754c561a0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:09.045762 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bae456d-0748-4885-9bfa-7d53887d8d33 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:09.046234 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bae456d-0748-4885-9bfa-7d53887d8d33 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:09.046946 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bae456d-0748-4885-9bfa-7d53887d8d33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:09.102334 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-1bae456d-0748-4885-9bfa-7d53887d8d33 None None] MFA Rules not processed for user `d311fc1f82124819b509f766fa432e2c`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:09.103563 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-1bae456d-0748-4885-9bfa-7d53887d8d33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:09.150924 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b114ec-7698-4278-aac6-948754c561a0 service neutron] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:09.152104 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 691/1402] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:09 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:09.177904 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895b0078-0273-4458-901f-cbc7b726082a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:09.178756 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895b0078-0273-4458-901f-cbc7b726082a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:09.178945 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895b0078-0273-4458-901f-cbc7b726082a admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:09.179280 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895b0078-0273-4458-901f-cbc7b726082a admin admin] PATH_INFO: `/v3/users/931e14007f454ec9be4b44c39b3a9e22` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:09.180901 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-895b0078-0273-4458-901f-cbc7b726082a admin admin] RBAC: Authorizing `identity:delete_user(user_id=931e14007f454ec9be4b44c39b3a9e22)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:09.183058 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-895b0078-0273-4458-901f-cbc7b726082a admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:09.341829 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-895b0078-0273-4458-901f-cbc7b726082a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '931e14007f454ec9be4b44c39b3a9e22'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:09.357656 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-895b0078-0273-4458-901f-cbc7b726082a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '931e14007f454ec9be4b44c39b3a9e22'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:09.391130 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-895b0078-0273-4458-901f-cbc7b726082a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '931e14007f454ec9be4b44c39b3a9e22'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:09.393485 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-895b0078-0273-4458-901f-cbc7b726082a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '931e14007f454ec9be4b44c39b3a9e22'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:09.394547 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-895b0078-0273-4458-901f-cbc7b726082a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '931e14007f454ec9be4b44c39b3a9e22'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:09.425399 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 712/1403] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:24:08 2022] DELETE /identity/v3/users/931e14007f454ec9be4b44c39b3a9e22 => generated 0 bytes in 533 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:09.427192 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c346ab-0eee-4452-b4cb-f029d2793bb4 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:09.427480 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c346ab-0eee-4452-b4cb-f029d2793bb4 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:09.427757 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c346ab-0eee-4452-b4cb-f029d2793bb4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:09.485557 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-c8c346ab-0eee-4452-b4cb-f029d2793bb4 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:09.486812 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-c8c346ab-0eee-4452-b4cb-f029d2793bb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:09.570035 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:09.683000 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 692/1404] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 257 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:09.715844 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895b0078-0273-4458-901f-cbc7b726082a admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:09.719611 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-895b0078-0273-4458-901f-cbc7b726082a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:09.754756 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:09.880149 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895b0078-0273-4458-901f-cbc7b726082a admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:09.888241 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e77459a-b3b8-45d4-83c8-0202095e18b7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:09.888916 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e77459a-b3b8-45d4-83c8-0202095e18b7 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:09.889234 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e77459a-b3b8-45d4-83c8-0202095e18b7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:09.890460 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e77459a-b3b8-45d4-83c8-0202095e18b7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:09.947139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e77459a-b3b8-45d4-83c8-0202095e18b7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:09.948605 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e77459a-b3b8-45d4-83c8-0202095e18b7 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:09.999058 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 713/1405] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 284 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:24:12.887711 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f8f353-7fb4-4a41-9284-c34be32514d6 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:12.888824 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f8f353-7fb4-4a41-9284-c34be32514d6 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:12.889476 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f8f353-7fb4-4a41-9284-c34be32514d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:13.041994 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-b1f8f353-7fb4-4a41-9284-c34be32514d6 None None] MFA Rules not processed for user `7a3f8b5eb09f4ba29f4bd343bd91f6a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:13.045095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-b1f8f353-7fb4-4a41-9284-c34be32514d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:13.057259 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd9e58e-56b3-4e40-8882-2bfe9acc5102 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:13.057719 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd9e58e-56b3-4e40-8882-2bfe9acc5102 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:13.057786 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd9e58e-56b3-4e40-8882-2bfe9acc5102 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:13.115594 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-7dd9e58e-56b3-4e40-8882-2bfe9acc5102 None None] MFA Rules not processed for user `99bcd22ff73b4526a54cf0e76f509436`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:13.118089 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-7dd9e58e-56b3-4e40-8882-2bfe9acc5102 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:13.237216 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 693/1406] 10.222.0.42 () {58 vars in 1009 bytes} [Thu Oct 27 01:24:12 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 351 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:24:13.320055 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 714/1407] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:13 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 263 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:13.411428 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f8f353-7fb4-4a41-9284-c34be32514d6 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:13.415459 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-b1f8f353-7fb4-4a41-9284-c34be32514d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:13.438746 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f8f353-7fb4-4a41-9284-c34be32514d6 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:13.448188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0434d17f-4aff-4de9-82e9-758b29adc607 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:13.449192 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0434d17f-4aff-4de9-82e9-758b29adc607 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:13.449793 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0434d17f-4aff-4de9-82e9-758b29adc607 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:13.450209 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0434d17f-4aff-4de9-82e9-758b29adc607 admin admin] PATH_INFO: `/v3/projects/1de72905e70a4b7ab0bc68459daac163` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:13.451961 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0434d17f-4aff-4de9-82e9-758b29adc607 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1de72905e70a4b7ab0bc68459daac163)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:13.454696 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0434d17f-4aff-4de9-82e9-758b29adc607 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:13.546614 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-0434d17f-4aff-4de9-82e9-758b29adc607 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1de72905e70a4b7ab0bc68459daac163', 'request_id': 'req-0434d17f-4aff-4de9-82e9-758b29adc607'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:13.559130 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd9e58e-56b3-4e40-8882-2bfe9acc5102 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:13.563674 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-7dd9e58e-56b3-4e40-8882-2bfe9acc5102 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:13.563787 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-0434d17f-4aff-4de9-82e9-758b29adc607 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1de72905e70a4b7ab0bc68459daac163', 'request_id': 'req-0434d17f-4aff-4de9-82e9-758b29adc607'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:13.577031 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-0434d17f-4aff-4de9-82e9-758b29adc607 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1de72905e70a4b7ab0bc68459daac163', 'request_id': 'req-0434d17f-4aff-4de9-82e9-758b29adc607'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:13.606731 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-0434d17f-4aff-4de9-82e9-758b29adc607 admin admin] The token cache is being invalidate because project 1de72905e70a4b7ab0bc68459daac163 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:24:13.607225 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-0434d17f-4aff-4de9-82e9-758b29adc607 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:13.612611 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-0434d17f-4aff-4de9-82e9-758b29adc607 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:13.617372 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 694/1408] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:24:13 2022] DELETE /identity/v3/projects/1de72905e70a4b7ab0bc68459daac163 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:13.622499 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d7e6a2-0d43-43e0-a497-d02dbce47e53 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:13.623192 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d7e6a2-0d43-43e0-a497-d02dbce47e53 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:13.623934 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d7e6a2-0d43-43e0-a497-d02dbce47e53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:13.674177 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-a1d7e6a2-0d43-43e0-a497-d02dbce47e53 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:13.682395 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-a1d7e6a2-0d43-43e0-a497-d02dbce47e53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:13.713167 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:13.850273 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 695/1409] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 229 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:13.858244 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540fd9b8-6822-4bfc-a7ec-66909837352c None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:13.858700 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540fd9b8-6822-4bfc-a7ec-66909837352c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:13.859209 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540fd9b8-6822-4bfc-a7ec-66909837352c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:13.906179 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-540fd9b8-6822-4bfc-a7ec-66909837352c None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:13.909925 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-540fd9b8-6822-4bfc-a7ec-66909837352c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:13.936346 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd9e58e-56b3-4e40-8882-2bfe9acc5102 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:13.938298 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 696/1410] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:24:13.960435 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540fd9b8-6822-4bfc-a7ec-66909837352c None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:14.008515 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540fd9b8-6822-4bfc-a7ec-66909837352c None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:14.022195 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc358d7-64f2-48aa-a090-52792ab3f4eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:14.023163 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc358d7-64f2-48aa-a090-52792ab3f4eb admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:14.028352 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc358d7-64f2-48aa-a090-52792ab3f4eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:14.028352 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc358d7-64f2-48aa-a090-52792ab3f4eb admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:14.028352 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc358d7-64f2-48aa-a090-52792ab3f4eb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:24:14.028352 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc358d7-64f2-48aa-a090-52792ab3f4eb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:14.029966 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc358d7-64f2-48aa-a090-52792ab3f4eb admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:14.060704 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 697/1411] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:24:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 118 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:24:14.067568 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc358d7-64f2-48aa-a090-52792ab3f4eb admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:14.085154 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc358d7-64f2-48aa-a090-52792ab3f4eb admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:14.105457 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68dea834-a600-43fb-81ce-d5472e6b11a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:14.106529 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68dea834-a600-43fb-81ce-d5472e6b11a8 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:14.107025 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68dea834-a600-43fb-81ce-d5472e6b11a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:14.107492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68dea834-a600-43fb-81ce-d5472e6b11a8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:14.108337 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68dea834-a600-43fb-81ce-d5472e6b11a8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:14.110513 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68dea834-a600-43fb-81ce-d5472e6b11a8 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:14.113951 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:14.191301 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 698/1412] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:24:14 2022] POST /identity/v3/projects => generated 381 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:24:14.192520 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68dea834-a600-43fb-81ce-d5472e6b11a8 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:14.199028 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-68dea834-a600-43fb-81ce-d5472e6b11a8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:14.225718 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3ae169-8306-470d-8407-3836e49404cf service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77f2cf9e0c3541f9bccfb7e213d88609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:14.227154 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3ae169-8306-470d-8407-3836e49404cf service swift] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:14.228524 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3ae169-8306-470d-8407-3836e49404cf service swift] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:14.230371 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3ae169-8306-470d-8407-3836e49404cf service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:14.241335 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-ac3ae169-8306-470d-8407-3836e49404cf service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:14.398173 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68dea834-a600-43fb-81ce-d5472e6b11a8 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:14.412348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1768e45f-d12a-4ad1-b940-9ee0e3009378 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:14.413200 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1768e45f-d12a-4ad1-b940-9ee0e3009378 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:14.414059 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1768e45f-d12a-4ad1-b940-9ee0e3009378 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:14.414420 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1768e45f-d12a-4ad1-b940-9ee0e3009378 admin admin] PATH_INFO: `/v3/users/1023c639b6bc4f448c30c7cee2765788` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:14.416014 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1768e45f-d12a-4ad1-b940-9ee0e3009378 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1023c639b6bc4f448c30c7cee2765788)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:14.418897 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1768e45f-d12a-4ad1-b940-9ee0e3009378 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:14.446794 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac3ae169-8306-470d-8407-3836e49404cf service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:14.448946 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac3ae169-8306-470d-8407-3836e49404cf service swift] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:14.483311 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 715/1413] 10.222.0.42 () {60 vars in 1384 bytes} [Thu Oct 27 01:24:13 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 926 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 27 01:24:14.484258 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3ae169-8306-470d-8407-3836e49404cf service swift] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:14.511014 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3ae169-8306-470d-8407-3836e49404cf service swift] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:14.529192 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2329346d-b3aa-4df0-aaaa-c9e4668835dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:14.529992 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2329346d-b3aa-4df0-aaaa-c9e4668835dc admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:14.530373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2329346d-b3aa-4df0-aaaa-c9e4668835dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:14.530765 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2329346d-b3aa-4df0-aaaa-c9e4668835dc admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:14.531505 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2329346d-b3aa-4df0-aaaa-c9e4668835dc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:14.533598 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2329346d-b3aa-4df0-aaaa-c9e4668835dc admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:14.558967 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1768e45f-d12a-4ad1-b940-9ee0e3009378 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1023c639b6bc4f448c30c7cee2765788'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:14.576913 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 716/1414] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:24:14 2022] POST /identity/v3/users => generated 291 bytes in 89 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:24:14.576913 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2329346d-b3aa-4df0-aaaa-c9e4668835dc admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:14.577177 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1768e45f-d12a-4ad1-b940-9ee0e3009378 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1023c639b6bc4f448c30c7cee2765788'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:14.579786 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1768e45f-d12a-4ad1-b940-9ee0e3009378 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1023c639b6bc4f448c30c7cee2765788'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:14.581158 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-2329346d-b3aa-4df0-aaaa-c9e4668835dc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:14.607868 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2329346d-b3aa-4df0-aaaa-c9e4668835dc admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:14.614751 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1768e45f-d12a-4ad1-b940-9ee0e3009378 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1023c639b6bc4f448c30c7cee2765788'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:14.617756 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e42f87ed-0e8d-4962-9b2d-f80c34b7d39c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:14.618547 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e42f87ed-0e8d-4962-9b2d-f80c34b7d39c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:14.618926 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e42f87ed-0e8d-4962-9b2d-f80c34b7d39c admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:14.619306 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e42f87ed-0e8d-4962-9b2d-f80c34b7d39c admin admin] PATH_INFO: `/v3/users/c57e9aa111044db39778f796f17b4ead` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:14.621779 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e42f87ed-0e8d-4962-9b2d-f80c34b7d39c admin admin] RBAC: Authorizing `identity:delete_user(user_id=c57e9aa111044db39778f796f17b4ead)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:14.623958 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e42f87ed-0e8d-4962-9b2d-f80c34b7d39c admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:14.627115 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:14.635243 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1768e45f-d12a-4ad1-b940-9ee0e3009378 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1023c639b6bc4f448c30c7cee2765788'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:14.650533 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 699/1415] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:24:14 2022] DELETE /identity/v3/users/1023c639b6bc4f448c30c7cee2765788 => generated 0 bytes in 458 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:14.654911 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1768e45f-d12a-4ad1-b940-9ee0e3009378 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:14.656021 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-1768e45f-d12a-4ad1-b940-9ee0e3009378 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:14.725598 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-e42f87ed-0e8d-4962-9b2d-f80c34b7d39c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c57e9aa111044db39778f796f17b4ead'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:14.745762 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-e42f87ed-0e8d-4962-9b2d-f80c34b7d39c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c57e9aa111044db39778f796f17b4ead'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:14.789022 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-e42f87ed-0e8d-4962-9b2d-f80c34b7d39c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c57e9aa111044db39778f796f17b4ead'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:14.793470 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-e42f87ed-0e8d-4962-9b2d-f80c34b7d39c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c57e9aa111044db39778f796f17b4ead'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:14.797634 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-e42f87ed-0e8d-4962-9b2d-f80c34b7d39c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c57e9aa111044db39778f796f17b4ead'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:14.840076 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 717/1416] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:24:14 2022] DELETE /identity/v3/users/c57e9aa111044db39778f796f17b4ead => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:14.841537 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8781d12f-2ce9-4e86-b119-dbb84ba1d046 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:14.847890 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8781d12f-2ce9-4e86-b119-dbb84ba1d046 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:14.848318 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8781d12f-2ce9-4e86-b119-dbb84ba1d046 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:14.897856 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1768e45f-d12a-4ad1-b940-9ee0e3009378 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:14.906143 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-8781d12f-2ce9-4e86-b119-dbb84ba1d046 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:14.907849 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-8781d12f-2ce9-4e86-b119-dbb84ba1d046 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:14.995767 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:15.050050 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591ed139-e7bb-436f-9843-2c10d10a18a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:15.052215 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591ed139-e7bb-436f-9843-2c10d10a18a3 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:15.053026 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591ed139-e7bb-436f-9843-2c10d10a18a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:15.053486 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591ed139-e7bb-436f-9843-2c10d10a18a3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:15.054287 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-591ed139-e7bb-436f-9843-2c10d10a18a3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:15.057291 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-591ed139-e7bb-436f-9843-2c10d10a18a3 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:15.075044 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 700/1417] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:14 2022] GET /identity/v3/roles => generated 1420 bytes in 424 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:15.076322 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591ed139-e7bb-436f-9843-2c10d10a18a3 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:15.078935 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-591ed139-e7bb-436f-9843-2c10d10a18a3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:15.084420 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 718/1418] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 243 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:15.085896 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583bbde9-336a-4cc9-84dc-b72fa2b4e385 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:15.086181 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583bbde9-336a-4cc9-84dc-b72fa2b4e385 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:15.086454 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583bbde9-336a-4cc9-84dc-b72fa2b4e385 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:15.105620 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591ed139-e7bb-436f-9843-2c10d10a18a3 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:15.116560 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-632ea1e9-a7ef-4ac1-ba55-a4df2a9de4d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:15.117437 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632ea1e9-a7ef-4ac1-ba55-a4df2a9de4d3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:15.119410 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632ea1e9-a7ef-4ac1-ba55-a4df2a9de4d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:15.119677 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632ea1e9-a7ef-4ac1-ba55-a4df2a9de4d3 admin admin] PATH_INFO: `/v3/users/7bbfb93f65f64b9b9fd686cf13776c63` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:15.122013 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-583bbde9-336a-4cc9-84dc-b72fa2b4e385 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:15.122865 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-632ea1e9-a7ef-4ac1-ba55-a4df2a9de4d3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7bbfb93f65f64b9b9fd686cf13776c63)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:15.124990 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-583bbde9-336a-4cc9-84dc-b72fa2b4e385 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:15.126130 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-632ea1e9-a7ef-4ac1-ba55-a4df2a9de4d3 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:15.155570 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 719/1419] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 70 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:15.156838 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583bbde9-336a-4cc9-84dc-b72fa2b4e385 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:15.159220 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-583bbde9-336a-4cc9-84dc-b72fa2b4e385 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:15.173717 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583bbde9-336a-4cc9-84dc-b72fa2b4e385 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:15.186898 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3ee9b2-7fb5-4e59-a07e-140385904628 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:15.188212 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3ee9b2-7fb5-4e59-a07e-140385904628 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:15.189014 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3ee9b2-7fb5-4e59-a07e-140385904628 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:15.189526 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3ee9b2-7fb5-4e59-a07e-140385904628 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:15.190756 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b3ee9b2-7fb5-4e59-a07e-140385904628 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:15.196464 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b3ee9b2-7fb5-4e59-a07e-140385904628 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:15.224869 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 720/1420] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:15 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:15.226441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-632ea1e9-a7ef-4ac1-ba55-a4df2a9de4d3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7bbfb93f65f64b9b9fd686cf13776c63'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:15.228437 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3ee9b2-7fb5-4e59-a07e-140385904628 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:15.242453 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-632ea1e9-a7ef-4ac1-ba55-a4df2a9de4d3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7bbfb93f65f64b9b9fd686cf13776c63'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:15.243848 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-632ea1e9-a7ef-4ac1-ba55-a4df2a9de4d3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7bbfb93f65f64b9b9fd686cf13776c63'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:15.254429 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3ee9b2-7fb5-4e59-a07e-140385904628 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:15.267235 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9da75fe7-4b11-421d-8f52-c6f4322a0583 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:15.269717 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da75fe7-4b11-421d-8f52-c6f4322a0583 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:15.271222 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da75fe7-4b11-421d-8f52-c6f4322a0583 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:15.271887 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da75fe7-4b11-421d-8f52-c6f4322a0583 admin admin] PATH_INFO: `/v3/projects/bf79533a12114d758249ef92f8a11a71/users/fc808f90939541c7a0492c80cca2f8d7/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:15.274270 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-632ea1e9-a7ef-4ac1-ba55-a4df2a9de4d3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7bbfb93f65f64b9b9fd686cf13776c63'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:15.295563 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-632ea1e9-a7ef-4ac1-ba55-a4df2a9de4d3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7bbfb93f65f64b9b9fd686cf13776c63'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:15.312778 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 701/1421] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:24:15 2022] DELETE /identity/v3/users/7bbfb93f65f64b9b9fd686cf13776c63 => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:15.314311 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9da75fe7-4b11-421d-8f52-c6f4322a0583 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bf79533a12114d758249ef92f8a11a71, user_id=fc808f90939541c7a0492c80cca2f8d7, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:15.318767 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbde526-598c-4ccd-a193-25f8313ba5e3 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:15.320431 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbde526-598c-4ccd-a193-25f8313ba5e3 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:15.321038 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbde526-598c-4ccd-a193-25f8313ba5e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:15.324647 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9da75fe7-4b11-421d-8f52-c6f4322a0583 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:15.360015 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 721/1422] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:24:15 2022] PUT /identity/v3/projects/bf79533a12114d758249ef92f8a11a71/users/fc808f90939541c7a0492c80cca2f8d7/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:15.362144 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9da75fe7-4b11-421d-8f52-c6f4322a0583 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:15.372310 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-9da75fe7-4b11-421d-8f52-c6f4322a0583 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:15.375361 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-fdbde526-598c-4ccd-a193-25f8313ba5e3 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:15.386039 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-fdbde526-598c-4ccd-a193-25f8313ba5e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:15.400681 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:15.561906 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9da75fe7-4b11-421d-8f52-c6f4322a0583 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:15.567249 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723526ad-bc62-4209-9163-191efb9a0623 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:15.569567 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723526ad-bc62-4209-9163-191efb9a0623 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:15.569965 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723526ad-bc62-4209-9163-191efb9a0623 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:15.570243 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723526ad-bc62-4209-9163-191efb9a0623 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:15.573322 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-723526ad-bc62-4209-9163-191efb9a0623 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:15.607815 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 702/1423] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 292 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:15.609247 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3fa86d-11f4-48e5-a0d1-e5baa494f8a7 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:15.609530 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3fa86d-11f4-48e5-a0d1-e5baa494f8a7 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:15.609809 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3fa86d-11f4-48e5-a0d1-e5baa494f8a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:15.719091 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-1e3fa86d-11f4-48e5-a0d1-e5baa494f8a7 None None] MFA Rules not processed for user `fc808f90939541c7a0492c80cca2f8d7`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:15.720385 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-1e3fa86d-11f4-48e5-a0d1-e5baa494f8a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:15.780883 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723526ad-bc62-4209-9163-191efb9a0623 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:15.791072 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723526ad-bc62-4209-9163-191efb9a0623 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:15.852432 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 722/1424] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 491 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:24:15.855780 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723526ad-bc62-4209-9163-191efb9a0623 service neutron] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:15.882225 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723526ad-bc62-4209-9163-191efb9a0623 service neutron] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:15.887142 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-002e1742-ad62-421e-9944-61ba3f5d977e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:15.888878 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002e1742-ad62-421e-9944-61ba3f5d977e service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:15.889558 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002e1742-ad62-421e-9944-61ba3f5d977e service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:15.890202 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002e1742-ad62-421e-9944-61ba3f5d977e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:15.892528 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-002e1742-ad62-421e-9944-61ba3f5d977e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:15.932323 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 703/1425] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:15 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 324 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:15.933341 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3fa86d-11f4-48e5-a0d1-e5baa494f8a7 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:15.936282 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-002e1742-ad62-421e-9944-61ba3f5d977e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:15.943024 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-002e1742-ad62-421e-9944-61ba3f5d977e service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:15.953762 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3fa86d-11f4-48e5-a0d1-e5baa494f8a7 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:15.964706 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb9d04b-ffe9-4c74-a9f6-99c4088750f8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:15.966464 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb9d04b-ffe9-4c74-a9f6-99c4088750f8 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:15.967244 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb9d04b-ffe9-4c74-a9f6-99c4088750f8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:15.967699 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb9d04b-ffe9-4c74-a9f6-99c4088750f8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:15.980435 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 723/1426] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:24:15.984926 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-002e1742-ad62-421e-9944-61ba3f5d977e service neutron] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:15.990688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-002e1742-ad62-421e-9944-61ba3f5d977e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:15.994137 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb9d04b-ffe9-4c74-a9f6-99c4088750f8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:15.996657 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb9d04b-ffe9-4c74-a9f6-99c4088750f8 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:16.029206 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 704/1427] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:24:16.073639 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-002e1742-ad62-421e-9944-61ba3f5d977e service neutron] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:16.102265 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f53782-ec52-485c-bdbc-2b4b2810c537 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:16.104668 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f53782-ec52-485c-bdbc-2b4b2810c537 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:16.105228 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f53782-ec52-485c-bdbc-2b4b2810c537 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:16.106992 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f53782-ec52-485c-bdbc-2b4b2810c537 admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:16.108843 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f53782-ec52-485c-bdbc-2b4b2810c537 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:16.111012 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f53782-ec52-485c-bdbc-2b4b2810c537 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:16.177580 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 724/1428] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:24:15 2022] POST /identity/v3/projects => generated 379 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:24:16.183008 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb9d04b-ffe9-4c74-a9f6-99c4088750f8 service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:16.213093 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb9d04b-ffe9-4c74-a9f6-99c4088750f8 service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:16.467235 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589ae97f-8c59-437f-bcc7-ee5f3fe3b5c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:16.473624 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589ae97f-8c59-437f-bcc7-ee5f3fe3b5c7 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:16.474390 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589ae97f-8c59-437f-bcc7-ee5f3fe3b5c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:16.474664 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589ae97f-8c59-437f-bcc7-ee5f3fe3b5c7 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:16.475376 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-589ae97f-8c59-437f-bcc7-ee5f3fe3b5c7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:16.477626 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-589ae97f-8c59-437f-bcc7-ee5f3fe3b5c7 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:16.546025 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 705/1429] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:24:16 2022] POST /identity/v3/users => generated 296 bytes in 363 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:24:16.551001 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f53782-ec52-485c-bdbc-2b4b2810c537 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:16.569641 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f53782-ec52-485c-bdbc-2b4b2810c537 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:16.598202 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50aa2447-4469-4523-bc97-ff4ef381aff3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:16.598856 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50aa2447-4469-4523-bc97-ff4ef381aff3 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:16.599138 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50aa2447-4469-4523-bc97-ff4ef381aff3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:16.599410 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50aa2447-4469-4523-bc97-ff4ef381aff3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:16.616207 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50aa2447-4469-4523-bc97-ff4ef381aff3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:16.621755 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50aa2447-4469-4523-bc97-ff4ef381aff3 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:16.642164 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:16.664657 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 725/1430] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:16 2022] GET /identity/v3/roles => generated 1420 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:16.667145 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589ae97f-8c59-437f-bcc7-ee5f3fe3b5c7 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:16.685052 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589ae97f-8c59-437f-bcc7-ee5f3fe3b5c7 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:16.704868 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a6a797-e11f-4329-8140-97299b53d6d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:16.705661 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a6a797-e11f-4329-8140-97299b53d6d0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:16.705947 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a6a797-e11f-4329-8140-97299b53d6d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:16.706224 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a6a797-e11f-4329-8140-97299b53d6d0 admin admin] PATH_INFO: `/v3/projects/0eb5ac85ccde4e08bdd70cd477fc0bb3/users/2ca19a201243468583cf1b44beacd989/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:16.752136 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79a6a797-e11f-4329-8140-97299b53d6d0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0eb5ac85ccde4e08bdd70cd477fc0bb3, user_id=2ca19a201243468583cf1b44beacd989, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:16.758257 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79a6a797-e11f-4329-8140-97299b53d6d0 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:16.768372 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:16.797342 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 706/1431] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:24:16 2022] PUT /identity/v3/projects/0eb5ac85ccde4e08bdd70cd477fc0bb3/users/2ca19a201243468583cf1b44beacd989/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:16.802056 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50aa2447-4469-4523-bc97-ff4ef381aff3 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:16.829769 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50aa2447-4469-4523-bc97-ff4ef381aff3 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:16.888733 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f695f774-8f10-4bc0-a434-bcf1756a0c00 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:16.889018 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f695f774-8f10-4bc0-a434-bcf1756a0c00 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:16.889352 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f695f774-8f10-4bc0-a434-bcf1756a0c00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:16.927999 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-f695f774-8f10-4bc0-a434-bcf1756a0c00 None None] MFA Rules not processed for user `4072417990c94b0488871332d28224ba`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:16.937645 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f695f774-8f10-4bc0-a434-bcf1756a0c00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:17.065695 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0984ba7b-5ab3-4257-a92c-e4791f7b642c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:17.068632 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0984ba7b-5ab3-4257-a92c-e4791f7b642c admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:17.073036 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0984ba7b-5ab3-4257-a92c-e4791f7b642c admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:17.073605 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0984ba7b-5ab3-4257-a92c-e4791f7b642c admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:17.074746 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0984ba7b-5ab3-4257-a92c-e4791f7b642c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:17.076736 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0984ba7b-5ab3-4257-a92c-e4791f7b642c admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:17.082089 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 707/1432] 10.222.0.13 () {58 vars in 1015 bytes} [Thu Oct 27 01:24:16 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:24:17.135359 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 726/1433] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:16 2022] GET /identity/v3/roles => generated 1420 bytes in 335 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:17.139841 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f695f774-8f10-4bc0-a434-bcf1756a0c00 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:17.160697 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f695f774-8f10-4bc0-a434-bcf1756a0c00 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:17.175431 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a32b46b-bf83-4d5f-b44b-a22fa5a8b74c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:17.178967 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a32b46b-bf83-4d5f-b44b-a22fa5a8b74c admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:17.179183 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a32b46b-bf83-4d5f-b44b-a22fa5a8b74c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:17.179468 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a32b46b-bf83-4d5f-b44b-a22fa5a8b74c admin admin] PATH_INFO: `/v3/projects/0eb5ac85ccde4e08bdd70cd477fc0bb3/users/2ca19a201243468583cf1b44beacd989/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:17.185916 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a32b46b-bf83-4d5f-b44b-a22fa5a8b74c admin admin] RBAC: Authorizing `identity:create_grant(project_id=0eb5ac85ccde4e08bdd70cd477fc0bb3, user_id=2ca19a201243468583cf1b44beacd989, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:17.192088 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a32b46b-bf83-4d5f-b44b-a22fa5a8b74c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:17.230291 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 708/1434] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:24:17 2022] PUT /identity/v3/projects/0eb5ac85ccde4e08bdd70cd477fc0bb3/users/2ca19a201243468583cf1b44beacd989/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 91 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:17.241469 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd33934-8914-4dd4-8ffe-5c604f548063 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:17.242742 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd33934-8914-4dd4-8ffe-5c604f548063 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:17.243989 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd33934-8914-4dd4-8ffe-5c604f548063 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:17.342686 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a32b46b-bf83-4d5f-b44b-a22fa5a8b74c admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:17.360843 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-5dd33934-8914-4dd4-8ffe-5c604f548063 None None] MFA Rules not processed for user `2ca19a201243468583cf1b44beacd989`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:17.363978 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-5dd33934-8914-4dd4-8ffe-5c604f548063 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:17.497592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a32b46b-bf83-4d5f-b44b-a22fa5a8b74c admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:17.509603 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58cb2fa8-8cde-4f47-a4c7-a3642afdd95c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:17.511010 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58cb2fa8-8cde-4f47-a4c7-a3642afdd95c service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:17.512072 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58cb2fa8-8cde-4f47-a4c7-a3642afdd95c service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:17.512184 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58cb2fa8-8cde-4f47-a4c7-a3642afdd95c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:17.558566 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58cb2fa8-8cde-4f47-a4c7-a3642afdd95c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:17.560497 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58cb2fa8-8cde-4f47-a4c7-a3642afdd95c service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:17.571422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 727/1435] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:17 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 331 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:17.594469 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd33934-8914-4dd4-8ffe-5c604f548063 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:17.600719 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 709/1436] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:17 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 258 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:24:17.601633 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-5dd33934-8914-4dd4-8ffe-5c604f548063 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:17.794686 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd33934-8914-4dd4-8ffe-5c604f548063 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:17.842678 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03cc8583-9953-44cd-8b35-e08c32861c50 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:17.844196 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03cc8583-9953-44cd-8b35-e08c32861c50 service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:17.846192 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03cc8583-9953-44cd-8b35-e08c32861c50 service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:17.846465 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03cc8583-9953-44cd-8b35-e08c32861c50 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:17.870815 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664ba3f8-2081-4c1f-8de1-2c84a610cba7 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:17.871173 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664ba3f8-2081-4c1f-8de1-2c84a610cba7 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:17.871442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664ba3f8-2081-4c1f-8de1-2c84a610cba7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:17.885331 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03cc8583-9953-44cd-8b35-e08c32861c50 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:17.886943 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03cc8583-9953-44cd-8b35-e08c32861c50 service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:17.946688 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-664ba3f8-2081-4c1f-8de1-2c84a610cba7 None None] MFA Rules not processed for user `cca36a277e3643a498b27e3c5d5e14f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:17.947398 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 728/1437] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:17 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:24:17.948848 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-664ba3f8-2081-4c1f-8de1-2c84a610cba7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:18.164263 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03cc8583-9953-44cd-8b35-e08c32861c50 service cinder] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:18.175943 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03cc8583-9953-44cd-8b35-e08c32861c50 service cinder] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:18.188017 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e34168c3-e498-4bf2-a0cc-0394525ea93b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:18.190168 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34168c3-e498-4bf2-a0cc-0394525ea93b service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:18.194030 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34168c3-e498-4bf2-a0cc-0394525ea93b service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:18.194655 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34168c3-e498-4bf2-a0cc-0394525ea93b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:18.196295 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 710/1438] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:17 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 327 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:18.336822 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:18.536654 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e34168c3-e498-4bf2-a0cc-0394525ea93b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:18.546399 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e34168c3-e498-4bf2-a0cc-0394525ea93b service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:18.579317 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 729/1439] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:18 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 415 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:24:19.134340 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664ba3f8-2081-4c1f-8de1-2c84a610cba7 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:19.135759 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-664ba3f8-2081-4c1f-8de1-2c84a610cba7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:19.291662 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:19.355175 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664ba3f8-2081-4c1f-8de1-2c84a610cba7 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:19.364225 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d679075-c4df-4a1c-868b-7bf5510277c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:19.364876 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d679075-c4df-4a1c-868b-7bf5510277c5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:19.365224 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d679075-c4df-4a1c-868b-7bf5510277c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:19.365562 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d679075-c4df-4a1c-868b-7bf5510277c5 admin admin] PATH_INFO: `/v3/projects/9e496b7490434d1cb34d22fe39c26e54` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:19.370463 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d679075-c4df-4a1c-868b-7bf5510277c5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9e496b7490434d1cb34d22fe39c26e54)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:19.373022 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d679075-c4df-4a1c-868b-7bf5510277c5 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:19.454377 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e34168c3-e498-4bf2-a0cc-0394525ea93b service cinder] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:19.457693 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-e34168c3-e498-4bf2-a0cc-0394525ea93b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:19.484434 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e34168c3-e498-4bf2-a0cc-0394525ea93b service cinder] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:19.507949 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2d679075-c4df-4a1c-868b-7bf5510277c5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9e496b7490434d1cb34d22fe39c26e54', 'request_id': 'req-2d679075-c4df-4a1c-868b-7bf5510277c5'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:19.539825 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2d679075-c4df-4a1c-868b-7bf5510277c5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9e496b7490434d1cb34d22fe39c26e54', 'request_id': 'req-2d679075-c4df-4a1c-868b-7bf5510277c5'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:19.588224 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2d679075-c4df-4a1c-868b-7bf5510277c5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9e496b7490434d1cb34d22fe39c26e54', 'request_id': 'req-2d679075-c4df-4a1c-868b-7bf5510277c5'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:19.635660 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2d679075-c4df-4a1c-868b-7bf5510277c5 admin admin] The token cache is being invalidate because project 9e496b7490434d1cb34d22fe39c26e54 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:24:19.636272 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2d679075-c4df-4a1c-868b-7bf5510277c5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:19.637192 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2d679075-c4df-4a1c-868b-7bf5510277c5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:19.642455 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 711/1440] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:24:19 2022] DELETE /identity/v3/projects/9e496b7490434d1cb34d22fe39c26e54 => generated 0 bytes in 505 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:19.644494 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dab30a3-1fbf-4f40-a9e3-abe14627c1b3 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:19.644780 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dab30a3-1fbf-4f40-a9e3-abe14627c1b3 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:19.645090 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dab30a3-1fbf-4f40-a9e3-abe14627c1b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:19.685037 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0889580-fa85-4bdf-831e-387cfc70c199 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:19.686217 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0889580-fa85-4bdf-831e-387cfc70c199 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:19.686681 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0889580-fa85-4bdf-831e-387cfc70c199 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:19.687101 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0889580-fa85-4bdf-831e-387cfc70c199 admin admin] PATH_INFO: `/v3/projects/1dbe731cc92548a49f9ee0d6abd4ad3e` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:19.688778 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0889580-fa85-4bdf-831e-387cfc70c199 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1dbe731cc92548a49f9ee0d6abd4ad3e)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:19.691269 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0889580-fa85-4bdf-831e-387cfc70c199 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:19.736877 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-0dab30a3-1fbf-4f40-a9e3-abe14627c1b3 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:19.748338 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0dab30a3-1fbf-4f40-a9e3-abe14627c1b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:19.820343 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 712/1441] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:19.822536 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d0c996-c54a-4632-9a20-be5f8c90f7d4 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:19.823143 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d0c996-c54a-4632-9a20-be5f8c90f7d4 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:19.823143 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d0c996-c54a-4632-9a20-be5f8c90f7d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:19.859846 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-f0889580-fa85-4bdf-831e-387cfc70c199 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1dbe731cc92548a49f9ee0d6abd4ad3e', 'request_id': 'req-f0889580-fa85-4bdf-831e-387cfc70c199'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:19.873901 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-f0889580-fa85-4bdf-831e-387cfc70c199 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1dbe731cc92548a49f9ee0d6abd4ad3e', 'request_id': 'req-f0889580-fa85-4bdf-831e-387cfc70c199'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:19.894435 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-f0889580-fa85-4bdf-831e-387cfc70c199 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1dbe731cc92548a49f9ee0d6abd4ad3e', 'request_id': 'req-f0889580-fa85-4bdf-831e-387cfc70c199'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:19.896623 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-09d0c996-c54a-4632-9a20-be5f8c90f7d4 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:19.897911 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-09d0c996-c54a-4632-9a20-be5f8c90f7d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:19.910610 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-f0889580-fa85-4bdf-831e-387cfc70c199 admin admin] The token cache is being invalidate because project 1dbe731cc92548a49f9ee0d6abd4ad3e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:24:19.911092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-f0889580-fa85-4bdf-831e-387cfc70c199 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:19.911873 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-f0889580-fa85-4bdf-831e-387cfc70c199 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:19.917763 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 730/1442] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:24:19 2022] DELETE /identity/v3/projects/1dbe731cc92548a49f9ee0d6abd4ad3e => generated 0 bytes in 460 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:24:19.920058 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfa5488-d2cf-4d2f-8b21-2f5a56f7ec19 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:19.921804 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfa5488-d2cf-4d2f-8b21-2f5a56f7ec19 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:19.921804 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfa5488-d2cf-4d2f-8b21-2f5a56f7ec19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:19.983914 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-3bfa5488-d2cf-4d2f-8b21-2f5a56f7ec19 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:19.988672 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-3bfa5488-d2cf-4d2f-8b21-2f5a56f7ec19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:19.989345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 713/1443] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:19.993936 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d0c996-c54a-4632-9a20-be5f8c90f7d4 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:20.023553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d0c996-c54a-4632-9a20-be5f8c90f7d4 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:20.031180 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67baada3-185f-4c83-9ac0-c07a8f5425fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:20.033098 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67baada3-185f-4c83-9ac0-c07a8f5425fc admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:20.033098 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67baada3-185f-4c83-9ac0-c07a8f5425fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:20.033098 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67baada3-185f-4c83-9ac0-c07a8f5425fc admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:20.035392 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67baada3-185f-4c83-9ac0-c07a8f5425fc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:24:20.036303 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67baada3-185f-4c83-9ac0-c07a8f5425fc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:20.038313 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67baada3-185f-4c83-9ac0-c07a8f5425fc admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:20.051829 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 731/1444] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:20.065413 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 714/1445] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:24:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:24:20.070837 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67baada3-185f-4c83-9ac0-c07a8f5425fc admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:20.071815 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268858f7-cc9e-49cb-ae43-208c629f50c8 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:20.072787 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268858f7-cc9e-49cb-ae43-208c629f50c8 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:20.077010 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268858f7-cc9e-49cb-ae43-208c629f50c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:20.088068 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67baada3-185f-4c83-9ac0-c07a8f5425fc admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:20.093910 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf765ef-3326-47fb-8e12-3c43e1c59773 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:20.094968 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf765ef-3326-47fb-8e12-3c43e1c59773 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:20.095356 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf765ef-3326-47fb-8e12-3c43e1c59773 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:20.096179 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf765ef-3326-47fb-8e12-3c43e1c59773 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:20.136619 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf765ef-3326-47fb-8e12-3c43e1c59773 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:20.152038 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf765ef-3326-47fb-8e12-3c43e1c59773 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:20.165989 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-268858f7-cc9e-49cb-ae43-208c629f50c8 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:20.171081 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-268858f7-cc9e-49cb-ae43-208c629f50c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:20.211694 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 732/1446] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:20.214406 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268858f7-cc9e-49cb-ae43-208c629f50c8 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:20.255210 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268858f7-cc9e-49cb-ae43-208c629f50c8 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:20.281750 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad265cca-2cc6-47d0-b5b1-4189452e5e81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:20.286057 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad265cca-2cc6-47d0-b5b1-4189452e5e81 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:20.286057 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad265cca-2cc6-47d0-b5b1-4189452e5e81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:20.286637 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad265cca-2cc6-47d0-b5b1-4189452e5e81 admin admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:20.291367 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad265cca-2cc6-47d0-b5b1-4189452e5e81 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:24:20.291708 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad265cca-2cc6-47d0-b5b1-4189452e5e81 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:20.294039 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 715/1447] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:24:20 2022] POST /identity/v3/projects => generated 385 bytes in 223 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:24:20.295476 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf765ef-3326-47fb-8e12-3c43e1c59773 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:20.299652 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad265cca-2cc6-47d0-b5b1-4189452e5e81 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:20.352403 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 733/1448] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:24:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:24:20.353567 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad265cca-2cc6-47d0-b5b1-4189452e5e81 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:20.401465 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad265cca-2cc6-47d0-b5b1-4189452e5e81 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:20.427890 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:20.508744 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:20.563633 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17261cb8-712e-4a85-a37b-032800bea57c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:20.567307 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17261cb8-712e-4a85-a37b-032800bea57c admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:20.567918 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17261cb8-712e-4a85-a37b-032800bea57c admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:20.568456 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17261cb8-712e-4a85-a37b-032800bea57c admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:20.571129 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17261cb8-712e-4a85-a37b-032800bea57c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:20.574172 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17261cb8-712e-4a85-a37b-032800bea57c admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:20.595377 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf765ef-3326-47fb-8e12-3c43e1c59773 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:20.615930 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b955c8f-f7a0-493d-a54b-ecb4a724c2fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:20.619211 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b955c8f-f7a0-493d-a54b-ecb4a724c2fb admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:20.624470 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b955c8f-f7a0-493d-a54b-ecb4a724c2fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:20.627058 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b955c8f-f7a0-493d-a54b-ecb4a724c2fb admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:20.633455 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b955c8f-f7a0-493d-a54b-ecb4a724c2fb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:20.640294 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b955c8f-f7a0-493d-a54b-ecb4a724c2fb admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:20.655649 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 734/1449] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:24:20 2022] POST /identity/v3/projects => generated 373 bytes in 300 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:24:20.657170 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17261cb8-712e-4a85-a37b-032800bea57c admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:20.720626 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 716/1450] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:24:20 2022] POST /identity/v3/users => generated 293 bytes in 426 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:24:20.729388 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b955c8f-f7a0-493d-a54b-ecb4a724c2fb admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:20.793748 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b955c8f-f7a0-493d-a54b-ecb4a724c2fb admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:20.960618 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17261cb8-712e-4a85-a37b-032800bea57c admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:20.995981 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4c467f-204b-4f6e-8f64-64c79ff2949b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:20.995981 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4c467f-204b-4f6e-8f64-64c79ff2949b admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:20.995981 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4c467f-204b-4f6e-8f64-64c79ff2949b admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:20.995981 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4c467f-204b-4f6e-8f64-64c79ff2949b admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:21.000269 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4c467f-204b-4f6e-8f64-64c79ff2949b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:21.002492 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4c467f-204b-4f6e-8f64-64c79ff2949b admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:21.043785 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad2d797-c4c8-4d05-8b77-e5992aae3e26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:21.044794 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad2d797-c4c8-4d05-8b77-e5992aae3e26 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:21.047148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad2d797-c4c8-4d05-8b77-e5992aae3e26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:21.047659 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad2d797-c4c8-4d05-8b77-e5992aae3e26 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:21.048655 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad2d797-c4c8-4d05-8b77-e5992aae3e26 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:21.072793 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad2d797-c4c8-4d05-8b77-e5992aae3e26 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:21.108485 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 717/1451] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:20 2022] GET /identity/v3/roles => generated 1420 bytes in 381 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:21.114382 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad2d797-c4c8-4d05-8b77-e5992aae3e26 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:21.118956 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 735/1452] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:24:20 2022] POST /identity/v3/users => generated 287 bytes in 461 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:24:21.120448 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4c467f-204b-4f6e-8f64-64c79ff2949b admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:21.146124 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4c467f-204b-4f6e-8f64-64c79ff2949b admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:21.161379 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad2d797-c4c8-4d05-8b77-e5992aae3e26 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:21.169046 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f4676c-2ab1-4a46-8c24-949196a03490 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:21.177887 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f4676c-2ab1-4a46-8c24-949196a03490 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:21.178958 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f4676c-2ab1-4a46-8c24-949196a03490 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:21.179417 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f4676c-2ab1-4a46-8c24-949196a03490 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:21.180074 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f4676c-2ab1-4a46-8c24-949196a03490 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:21.184033 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43890c80-3c67-4c3b-b40c-9be52a16de60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:21.187547 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43890c80-3c67-4c3b-b40c-9be52a16de60 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:21.189320 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43890c80-3c67-4c3b-b40c-9be52a16de60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:21.189836 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43890c80-3c67-4c3b-b40c-9be52a16de60 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:21.195427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43890c80-3c67-4c3b-b40c-9be52a16de60 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:21.201261 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43890c80-3c67-4c3b-b40c-9be52a16de60 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:21.211382 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f4676c-2ab1-4a46-8c24-949196a03490 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:21.230176 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 736/1453] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:21 2022] GET /identity/v3/roles => generated 1420 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 27 01:24:21.231447 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f4676c-2ab1-4a46-8c24-949196a03490 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:21.245370 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f4676c-2ab1-4a46-8c24-949196a03490 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:21.251572 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0971563a-c3b7-44be-bfe6-8c885d9ea715 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:21.252263 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0971563a-c3b7-44be-bfe6-8c885d9ea715 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:21.252654 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0971563a-c3b7-44be-bfe6-8c885d9ea715 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:21.253019 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0971563a-c3b7-44be-bfe6-8c885d9ea715 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:21.254697 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0971563a-c3b7-44be-bfe6-8c885d9ea715 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:21.256559 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0971563a-c3b7-44be-bfe6-8c885d9ea715 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:21.267778 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 718/1454] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:21 2022] GET /identity/v3/roles => generated 1420 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:21.270838 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43890c80-3c67-4c3b-b40c-9be52a16de60 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:21.271746 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 737/1455] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:21 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:21.275960 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0971563a-c3b7-44be-bfe6-8c885d9ea715 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:21.290531 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0971563a-c3b7-44be-bfe6-8c885d9ea715 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:21.295950 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5a6650-afc4-4d4b-9e25-6b23717b7ef5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:21.298192 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5a6650-afc4-4d4b-9e25-6b23717b7ef5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:21.298470 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5a6650-afc4-4d4b-9e25-6b23717b7ef5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:21.298792 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5a6650-afc4-4d4b-9e25-6b23717b7ef5 admin admin] PATH_INFO: `/v3/projects/a7e7557398ad411a92d3ec3f315b97bd/users/1c471d2db8994adeac8d2cfe82801e4a/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:21.318830 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43890c80-3c67-4c3b-b40c-9be52a16de60 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:21.322857 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd5a6650-afc4-4d4b-9e25-6b23717b7ef5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a7e7557398ad411a92d3ec3f315b97bd, user_id=1c471d2db8994adeac8d2cfe82801e4a, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:21.327636 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd5a6650-afc4-4d4b-9e25-6b23717b7ef5 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:21.336690 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5779186-5c04-41f6-97bf-56beb30b76c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:21.338952 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5779186-5c04-41f6-97bf-56beb30b76c3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:21.339284 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5779186-5c04-41f6-97bf-56beb30b76c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:21.339592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5779186-5c04-41f6-97bf-56beb30b76c3 admin admin] PATH_INFO: `/v3/projects/3990cfa81b9d46318a34bb862d243121/users/b6def6d4027a45eb8b40c3db72fda121/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:21.358858 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 738/1456] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:24:21 2022] PUT /identity/v3/projects/a7e7557398ad411a92d3ec3f315b97bd/users/1c471d2db8994adeac8d2cfe82801e4a/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:24:21.372351 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cc11ac-2507-4b18-9b1e-a12813289a11 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:21.372482 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cc11ac-2507-4b18-9b1e-a12813289a11 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:21.372717 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cc11ac-2507-4b18-9b1e-a12813289a11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:21.396797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5779186-5c04-41f6-97bf-56beb30b76c3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3990cfa81b9d46318a34bb862d243121, user_id=b6def6d4027a45eb8b40c3db72fda121, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:21.407983 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5779186-5c04-41f6-97bf-56beb30b76c3 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:21.477597 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 719/1457] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:24:21 2022] PUT /identity/v3/projects/3990cfa81b9d46318a34bb862d243121/users/b6def6d4027a45eb8b40c3db72fda121/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:21.490210 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bea24a-7b04-48d7-9fa1-1b8bb6e7c066 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:21.490668 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bea24a-7b04-48d7-9fa1-1b8bb6e7c066 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:21.491593 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bea24a-7b04-48d7-9fa1-1b8bb6e7c066 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:21.491996 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-26cc11ac-2507-4b18-9b1e-a12813289a11 None None] MFA Rules not processed for user `1c471d2db8994adeac8d2cfe82801e4a`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:21.493258 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-26cc11ac-2507-4b18-9b1e-a12813289a11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:21.615995 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-55bea24a-7b04-48d7-9fa1-1b8bb6e7c066 None None] MFA Rules not processed for user `b6def6d4027a45eb8b40c3db72fda121`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:21.618422 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-55bea24a-7b04-48d7-9fa1-1b8bb6e7c066 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:21.622585 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:21.718223 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 739/1458] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:21 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 347 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:21.847798 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 720/1459] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:21 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 359 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:21.853051 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26cc11ac-2507-4b18-9b1e-a12813289a11 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:22.053787 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26cc11ac-2507-4b18-9b1e-a12813289a11 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:22.070270 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0398d1e1-3a58-4b85-b7ae-2a77afec4013 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:22.071663 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0398d1e1-3a58-4b85-b7ae-2a77afec4013 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:22.073250 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0398d1e1-3a58-4b85-b7ae-2a77afec4013 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:22.073579 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0398d1e1-3a58-4b85-b7ae-2a77afec4013 admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:22.075568 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0398d1e1-3a58-4b85-b7ae-2a77afec4013 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:22.081884 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0398d1e1-3a58-4b85-b7ae-2a77afec4013 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:22.196395 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 740/1460] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:24:21 2022] POST /identity/v3/projects => generated 385 bytes in 345 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:24:22.200447 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55bea24a-7b04-48d7-9fa1-1b8bb6e7c066 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:22.213045 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0398d1e1-3a58-4b85-b7ae-2a77afec4013 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:22.217105 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-0398d1e1-3a58-4b85-b7ae-2a77afec4013 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:22.233826 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55bea24a-7b04-48d7-9fa1-1b8bb6e7c066 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:22.242039 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:22.432151 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a76ae9f-0036-48e0-bf2f-15a03a8c0c41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:22.432981 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a76ae9f-0036-48e0-bf2f-15a03a8c0c41 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:22.433297 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a76ae9f-0036-48e0-bf2f-15a03a8c0c41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:22.433570 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a76ae9f-0036-48e0-bf2f-15a03a8c0c41 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:22.434279 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a76ae9f-0036-48e0-bf2f-15a03a8c0c41 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:22.436218 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a76ae9f-0036-48e0-bf2f-15a03a8c0c41 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:22.443062 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0398d1e1-3a58-4b85-b7ae-2a77afec4013 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:22.473653 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0435fe0-214f-403e-9fe8-9dd11c89d2ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:22.474462 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0435fe0-214f-403e-9fe8-9dd11c89d2ea admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:22.474747 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0435fe0-214f-403e-9fe8-9dd11c89d2ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:22.475076 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0435fe0-214f-403e-9fe8-9dd11c89d2ea admin admin] PATH_INFO: `/v3/projects/21e86216310241f987e86f2eab46b3e4` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:22.476456 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0435fe0-214f-403e-9fe8-9dd11c89d2ea admin admin] RBAC: Authorizing `identity:delete_project(project_id=21e86216310241f987e86f2eab46b3e4)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:22.479571 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0435fe0-214f-403e-9fe8-9dd11c89d2ea admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:22.484988 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 721/1461] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:24:22 2022] POST /identity/v3/users => generated 293 bytes in 285 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:24:22.485948 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a76ae9f-0036-48e0-bf2f-15a03a8c0c41 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:22.488792 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-9a76ae9f-0036-48e0-bf2f-15a03a8c0c41 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:22.585161 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-b0435fe0-214f-403e-9fe8-9dd11c89d2ea admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '21e86216310241f987e86f2eab46b3e4', 'request_id': 'req-b0435fe0-214f-403e-9fe8-9dd11c89d2ea'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:22.604898 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-b0435fe0-214f-403e-9fe8-9dd11c89d2ea admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '21e86216310241f987e86f2eab46b3e4', 'request_id': 'req-b0435fe0-214f-403e-9fe8-9dd11c89d2ea'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:22.607976 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a76ae9f-0036-48e0-bf2f-15a03a8c0c41 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:22.624414 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-b0435fe0-214f-403e-9fe8-9dd11c89d2ea admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '21e86216310241f987e86f2eab46b3e4', 'request_id': 'req-b0435fe0-214f-403e-9fe8-9dd11c89d2ea'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:22.646956 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:22.654612 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-b0435fe0-214f-403e-9fe8-9dd11c89d2ea admin admin] The token cache is being invalidate because project 21e86216310241f987e86f2eab46b3e4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:24:22.658047 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-b0435fe0-214f-403e-9fe8-9dd11c89d2ea admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:22.662982 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-b0435fe0-214f-403e-9fe8-9dd11c89d2ea admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:22.671872 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 741/1462] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:24:22 2022] DELETE /identity/v3/projects/21e86216310241f987e86f2eab46b3e4 => generated 0 bytes in 457 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:22.674472 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0435fe0-214f-403e-9fe8-9dd11c89d2ea admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:22.678232 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-b0435fe0-214f-403e-9fe8-9dd11c89d2ea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:22.845906 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e47605c-e26c-4ed6-a713-fe267320a53d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:22.850973 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e47605c-e26c-4ed6-a713-fe267320a53d service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:22.851182 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0435fe0-214f-403e-9fe8-9dd11c89d2ea admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:22.851361 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e47605c-e26c-4ed6-a713-fe267320a53d service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:22.851539 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e47605c-e26c-4ed6-a713-fe267320a53d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:22.860453 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0e47605c-e26c-4ed6-a713-fe267320a53d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:22.868993 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c60e03-7b68-44e2-939b-b05a6e709c72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:22.870018 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c60e03-7b68-44e2-939b-b05a6e709c72 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:22.870018 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c60e03-7b68-44e2-939b-b05a6e709c72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:22.870259 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c60e03-7b68-44e2-939b-b05a6e709c72 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:22.871030 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c60e03-7b68-44e2-939b-b05a6e709c72 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:22.877764 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c60e03-7b68-44e2-939b-b05a6e709c72 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:22.911798 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 742/1463] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:22 2022] GET /identity/v3/roles => generated 1420 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:22.915855 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c60e03-7b68-44e2-939b-b05a6e709c72 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:22.921292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-44c60e03-7b68-44e2-939b-b05a6e709c72 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:22.923785 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:22.953444 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c60e03-7b68-44e2-939b-b05a6e709c72 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:22.970816 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f15d44-5e7a-4efa-802d-6c00ea502aea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:22.972172 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f15d44-5e7a-4efa-802d-6c00ea502aea admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:22.972718 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f15d44-5e7a-4efa-802d-6c00ea502aea admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:22.973195 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f15d44-5e7a-4efa-802d-6c00ea502aea admin admin] PATH_INFO: `/v3/users/53e3e7704920474bbda4485d33337ef7` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:22.974619 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f15d44-5e7a-4efa-802d-6c00ea502aea admin admin] RBAC: Authorizing `identity:delete_user(user_id=53e3e7704920474bbda4485d33337ef7)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:22.976732 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f15d44-5e7a-4efa-802d-6c00ea502aea admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:22.978025 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e47605c-e26c-4ed6-a713-fe267320a53d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:22.980063 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e47605c-e26c-4ed6-a713-fe267320a53d service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:22.994377 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 722/1464] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:22 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 509 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:24:22.995651 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e47605c-e26c-4ed6-a713-fe267320a53d service nova] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:23.010533 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e47605c-e26c-4ed6-a713-fe267320a53d service nova] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:23.016715 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0dcd86-7d4a-4b00-adb4-509cd175c92b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:23.017374 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0dcd86-7d4a-4b00-adb4-509cd175c92b admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:23.017649 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0dcd86-7d4a-4b00-adb4-509cd175c92b admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:23.017984 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0dcd86-7d4a-4b00-adb4-509cd175c92b admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:23.018562 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e0dcd86-7d4a-4b00-adb4-509cd175c92b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:23.022674 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e0dcd86-7d4a-4b00-adb4-509cd175c92b admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:23.069018 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 723/1465] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:22 2022] GET /identity/v3/roles => generated 1420 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:23.072484 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0dcd86-7d4a-4b00-adb4-509cd175c92b admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:23.106577 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-f9f15d44-5e7a-4efa-802d-6c00ea502aea admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '53e3e7704920474bbda4485d33337ef7'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:23.108539 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0dcd86-7d4a-4b00-adb4-509cd175c92b admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:23.125618 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-f9f15d44-5e7a-4efa-802d-6c00ea502aea admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '53e3e7704920474bbda4485d33337ef7'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:23.130374 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-affdf112-56c5-407b-a771-46bd4cba3633 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:23.132741 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affdf112-56c5-407b-a771-46bd4cba3633 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:23.134724 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affdf112-56c5-407b-a771-46bd4cba3633 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:23.139249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affdf112-56c5-407b-a771-46bd4cba3633 admin admin] PATH_INFO: `/v3/projects/f6ffdabbaf92472296a0dd36f6123ebb/users/83ead3aaced3494facc375590f6f68b8/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:23.159716 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-f9f15d44-5e7a-4efa-802d-6c00ea502aea admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '53e3e7704920474bbda4485d33337ef7'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:23.161478 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-f9f15d44-5e7a-4efa-802d-6c00ea502aea admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '53e3e7704920474bbda4485d33337ef7'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:23.163595 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-f9f15d44-5e7a-4efa-802d-6c00ea502aea admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '53e3e7704920474bbda4485d33337ef7'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:23.201719 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 743/1466] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:24:22 2022] DELETE /identity/v3/users/53e3e7704920474bbda4485d33337ef7 => generated 0 bytes in 286 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:23.207683 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-affdf112-56c5-407b-a771-46bd4cba3633 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f6ffdabbaf92472296a0dd36f6123ebb, user_id=83ead3aaced3494facc375590f6f68b8, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:23.214557 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-affdf112-56c5-407b-a771-46bd4cba3633 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:23.264460 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 724/1467] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:24:23 2022] PUT /identity/v3/projects/f6ffdabbaf92472296a0dd36f6123ebb/users/83ead3aaced3494facc375590f6f68b8/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:23.276172 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a3d58a-6532-43df-8167-43fa591d69dd None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:23.276403 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a3d58a-6532-43df-8167-43fa591d69dd None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:23.276747 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a3d58a-6532-43df-8167-43fa591d69dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:23.360938 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-28a3d58a-6532-43df-8167-43fa591d69dd None None] MFA Rules not processed for user `83ead3aaced3494facc375590f6f68b8`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:23.362276 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-28a3d58a-6532-43df-8167-43fa591d69dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:23.447678 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b93aec-8683-46fb-b471-63ad42eec90a None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:23.448308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b93aec-8683-46fb-b471-63ad42eec90a None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:23.448308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b93aec-8683-46fb-b471-63ad42eec90a None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:23.449958 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 725/1468] 10.222.0.42 () {54 vars in 921 bytes} [Thu Oct 27 01:24:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 27 01:24:23.509889 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:23.527701 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 744/1469] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:23 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 249 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:23.528718 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b93aec-8683-46fb-b471-63ad42eec90a None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:23.531253 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f9b93aec-8683-46fb-b471-63ad42eec90a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:23.567848 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6bdb25-069f-43a0-9160-e3c68d3fe24b None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:23.568230 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6bdb25-069f-43a0-9160-e3c68d3fe24b None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:23.568512 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6bdb25-069f-43a0-9160-e3c68d3fe24b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:23.626654 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-ff6bdb25-069f-43a0-9160-e3c68d3fe24b None None] MFA Rules not processed for user `51033d56f85f4282a49ded6338440677`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:23.629931 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-ff6bdb25-069f-43a0-9160-e3c68d3fe24b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:23.763434 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b93aec-8683-46fb-b471-63ad42eec90a None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:23.769569 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02123b17-dcad-4775-b9df-8f9901d7989d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:23.770775 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02123b17-dcad-4775-b9df-8f9901d7989d admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:23.771104 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02123b17-dcad-4775-b9df-8f9901d7989d admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:23.771370 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02123b17-dcad-4775-b9df-8f9901d7989d admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:23.772158 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02123b17-dcad-4775-b9df-8f9901d7989d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:23.776453 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02123b17-dcad-4775-b9df-8f9901d7989d admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:23.848882 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 745/1470] 10.222.0.42 () {58 vars in 1018 bytes} [Thu Oct 27 01:24:23 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 281 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:23.860321 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 726/1471] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:24:23 2022] POST /identity/v3/projects => generated 383 bytes in 332 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:24:23.866760 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6bdb25-069f-43a0-9160-e3c68d3fe24b None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:23.886198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02123b17-dcad-4775-b9df-8f9901d7989d admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:23.889089 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-02123b17-dcad-4775-b9df-8f9901d7989d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:23.919978 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6bdb25-069f-43a0-9160-e3c68d3fe24b None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:24.045858 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02123b17-dcad-4775-b9df-8f9901d7989d admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:24.061953 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d651094e-23b7-4398-816a-c6ab1dc178a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:24.067780 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d651094e-23b7-4398-816a-c6ab1dc178a3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:24.070061 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d651094e-23b7-4398-816a-c6ab1dc178a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:24.070558 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d651094e-23b7-4398-816a-c6ab1dc178a3 admin admin] PATH_INFO: `/v3/users/1c471d2db8994adeac8d2cfe82801e4a` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:24.074288 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d651094e-23b7-4398-816a-c6ab1dc178a3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1c471d2db8994adeac8d2cfe82801e4a)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:24.077989 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d651094e-23b7-4398-816a-c6ab1dc178a3 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:24.088904 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c895c9c-aac3-432b-9a9a-734c1cf6b281 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:24.089570 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c895c9c-aac3-432b-9a9a-734c1cf6b281 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:24.089988 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c895c9c-aac3-432b-9a9a-734c1cf6b281 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:24.090568 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c895c9c-aac3-432b-9a9a-734c1cf6b281 admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:24.091139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c895c9c-aac3-432b-9a9a-734c1cf6b281 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:24.093206 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c895c9c-aac3-432b-9a9a-734c1cf6b281 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:24.137702 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=94,keystone:INSERT=2,keystone:DELETE=4 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:24.156304 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 746/1472] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:24:23 2022] POST /identity/v3/users => generated 298 bytes in 288 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:24:24.158047 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c895c9c-aac3-432b-9a9a-734c1cf6b281 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:24.174298 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c895c9c-aac3-432b-9a9a-734c1cf6b281 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:24.181866 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c314d0-1bff-4ba9-95a5-50bc276d65c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:24.182707 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c314d0-1bff-4ba9-95a5-50bc276d65c4 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:24.183100 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c314d0-1bff-4ba9-95a5-50bc276d65c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:24.183418 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c314d0-1bff-4ba9-95a5-50bc276d65c4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:24.184848 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c314d0-1bff-4ba9-95a5-50bc276d65c4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:24.186188 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c314d0-1bff-4ba9-95a5-50bc276d65c4 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:24.208344 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 747/1473] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:24 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:24.213685 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c314d0-1bff-4ba9-95a5-50bc276d65c4 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:24.222439 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d651094e-23b7-4398-816a-c6ab1dc178a3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1c471d2db8994adeac8d2cfe82801e4a'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:24.246773 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c314d0-1bff-4ba9-95a5-50bc276d65c4 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:24.256239 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d651094e-23b7-4398-816a-c6ab1dc178a3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1c471d2db8994adeac8d2cfe82801e4a'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:24.257082 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d651094e-23b7-4398-816a-c6ab1dc178a3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1c471d2db8994adeac8d2cfe82801e4a'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:24.266126 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a450062-f795-4cb8-a956-b326da008a0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:24.266918 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a450062-f795-4cb8-a956-b326da008a0c admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:24.267196 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a450062-f795-4cb8-a956-b326da008a0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:24.267468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a450062-f795-4cb8-a956-b326da008a0c admin admin] PATH_INFO: `/v3/projects/8ad725c644134e899181ac2a445e8f28/users/812f282a9ee64d55b1dc9523b030d53d/roles/414286927fdd44688107ff5e10235acf` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:24.292557 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a450062-f795-4cb8-a956-b326da008a0c admin admin] RBAC: Authorizing `identity:create_grant(project_id=8ad725c644134e899181ac2a445e8f28, user_id=812f282a9ee64d55b1dc9523b030d53d, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:24.295194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d651094e-23b7-4398-816a-c6ab1dc178a3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1c471d2db8994adeac8d2cfe82801e4a'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:24.295776 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a450062-f795-4cb8-a956-b326da008a0c admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:24.316828 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d651094e-23b7-4398-816a-c6ab1dc178a3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1c471d2db8994adeac8d2cfe82801e4a'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:24.318589 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 748/1474] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:24:24 2022] PUT /identity/v3/projects/8ad725c644134e899181ac2a445e8f28/users/812f282a9ee64d55b1dc9523b030d53d/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:24.319808 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a450062-f795-4cb8-a956-b326da008a0c admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:24.322816 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-9a450062-f795-4cb8-a956-b326da008a0c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:24.340350 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 727/1475] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:24:23 2022] DELETE /identity/v3/users/1c471d2db8994adeac8d2cfe82801e4a => generated 0 bytes in 453 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:24.342931 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a64b24-fb40-47a5-9396-42dd833385f8 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:24.343218 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a64b24-fb40-47a5-9396-42dd833385f8 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:24.343498 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a64b24-fb40-47a5-9396-42dd833385f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:24.390786 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-c6a64b24-fb40-47a5-9396-42dd833385f8 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:24.392045 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-c6a64b24-fb40-47a5-9396-42dd833385f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:24.396332 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:24.422792 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a450062-f795-4cb8-a956-b326da008a0c admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:24.505612 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 728/1476] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:24.517354 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e775764-d9cf-4cb2-852e-08715f3b17d2 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:24.517786 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e775764-d9cf-4cb2-852e-08715f3b17d2 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:24.518242 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e775764-d9cf-4cb2-852e-08715f3b17d2 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:24.520492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 729/1477] 10.222.0.42 () {54 vars in 921 bytes} [Thu Oct 27 01:24:24 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:24:24.536549 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca4f053-a7f2-4cdc-b612-477ad7081441 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:24.537164 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca4f053-a7f2-4cdc-b612-477ad7081441 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:24.537447 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca4f053-a7f2-4cdc-b612-477ad7081441 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:24.537718 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca4f053-a7f2-4cdc-b612-477ad7081441 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:24.538594 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca4f053-a7f2-4cdc-b612-477ad7081441 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:24.540255 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca4f053-a7f2-4cdc-b612-477ad7081441 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:24.554828 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 749/1478] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:24 2022] GET /identity/v3/roles => generated 1420 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:24.555479 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e775764-d9cf-4cb2-852e-08715f3b17d2 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:24.573872 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e775764-d9cf-4cb2-852e-08715f3b17d2 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:24.580099 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144adba4-3417-4f3e-be4d-0fdd3f535958 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:24.580773 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144adba4-3417-4f3e-be4d-0fdd3f535958 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:24.581048 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144adba4-3417-4f3e-be4d-0fdd3f535958 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:24.581989 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144adba4-3417-4f3e-be4d-0fdd3f535958 admin admin] PATH_INFO: `/v3/projects/8ad725c644134e899181ac2a445e8f28/users/812f282a9ee64d55b1dc9523b030d53d/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:24.584589 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144adba4-3417-4f3e-be4d-0fdd3f535958 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8ad725c644134e899181ac2a445e8f28, user_id=812f282a9ee64d55b1dc9523b030d53d, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:24.588797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144adba4-3417-4f3e-be4d-0fdd3f535958 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:24.602468 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 730/1479] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:24:24 2022] PUT /identity/v3/projects/8ad725c644134e899181ac2a445e8f28/users/812f282a9ee64d55b1dc9523b030d53d/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:24.615720 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123f2e0d-f94f-4970-8f92-1e25437764d4 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:24.615923 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123f2e0d-f94f-4970-8f92-1e25437764d4 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:24.616746 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123f2e0d-f94f-4970-8f92-1e25437764d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:24.629429 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144adba4-3417-4f3e-be4d-0fdd3f535958 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:24.633614 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-144adba4-3417-4f3e-be4d-0fdd3f535958 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:24.663726 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-123f2e0d-f94f-4970-8f92-1e25437764d4 None None] MFA Rules not processed for user `812f282a9ee64d55b1dc9523b030d53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:24.667251 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-123f2e0d-f94f-4970-8f92-1e25437764d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:24.778498 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144adba4-3417-4f3e-be4d-0fdd3f535958 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:24.790667 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebda27a4-9ddf-4917-a04e-f445c6816551 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:24.791456 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebda27a4-9ddf-4917-a04e-f445c6816551 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:24.793548 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebda27a4-9ddf-4917-a04e-f445c6816551 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:24.794143 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebda27a4-9ddf-4917-a04e-f445c6816551 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:24.816721 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 750/1480] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:24 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:24.818671 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1b1aad-e90b-40f7-8cb7-c9cfbba32a29 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:24.819232 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1b1aad-e90b-40f7-8cb7-c9cfbba32a29 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:24.819699 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1b1aad-e90b-40f7-8cb7-c9cfbba32a29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:24.869791 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-4d1b1aad-e90b-40f7-8cb7-c9cfbba32a29 None None] MFA Rules not processed for user `51033d56f85f4282a49ded6338440677`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:24.875232 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-4d1b1aad-e90b-40f7-8cb7-c9cfbba32a29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:24.910911 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=92,keystone:DELETE=6,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:24.935684 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebda27a4-9ddf-4917-a04e-f445c6816551 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:24.937343 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebda27a4-9ddf-4917-a04e-f445c6816551 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:24.988917 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 731/1481] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:24:24.991277 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebda27a4-9ddf-4917-a04e-f445c6816551 service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:24.995006 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ebda27a4-9ddf-4917-a04e-f445c6816551 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:25.077500 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 751/1482] 10.222.0.42 () {58 vars in 1018 bytes} [Thu Oct 27 01:24:24 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 260 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:25.078739 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1b1aad-e90b-40f7-8cb7-c9cfbba32a29 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:25.080437 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-4d1b1aad-e90b-40f7-8cb7-c9cfbba32a29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:25.088711 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:25.134648 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebda27a4-9ddf-4917-a04e-f445c6816551 service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:25.141357 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6470e655-3c94-4a08-ba16-183fc271a2dc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:25.142646 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6470e655-3c94-4a08-ba16-183fc271a2dc service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:25.143075 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6470e655-3c94-4a08-ba16-183fc271a2dc service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:25.143492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6470e655-3c94-4a08-ba16-183fc271a2dc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:25.145710 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-6470e655-3c94-4a08-ba16-183fc271a2dc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:25.161679 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6470e655-3c94-4a08-ba16-183fc271a2dc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:25.163903 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6470e655-3c94-4a08-ba16-183fc271a2dc service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:25.164551 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1b1aad-e90b-40f7-8cb7-c9cfbba32a29 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:25.168543 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c9aa11-09d3-432d-89fe-033c9e5ec444 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:25.169257 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c9aa11-09d3-432d-89fe-033c9e5ec444 service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:25.169538 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c9aa11-09d3-432d-89fe-033c9e5ec444 service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:25.169808 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c9aa11-09d3-432d-89fe-033c9e5ec444 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:25.171924 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-c0c9aa11-09d3-432d-89fe-033c9e5ec444 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:25.179753 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 732/1483] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:24 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 190 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:24:25.255092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c9aa11-09d3-432d-89fe-033c9e5ec444 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:25.257351 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c9aa11-09d3-432d-89fe-033c9e5ec444 service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:25.257552 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6470e655-3c94-4a08-ba16-183fc271a2dc service nova] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:25.258986 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-6470e655-3c94-4a08-ba16-183fc271a2dc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:25.286139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 752/1484] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:25 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 208 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:24:25.297141 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6470e655-3c94-4a08-ba16-183fc271a2dc service nova] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:25.313603 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0377498-6c12-44b6-8dc1-26fb0def806e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:25.314533 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0377498-6c12-44b6-8dc1-26fb0def806e service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:25.314639 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0377498-6c12-44b6-8dc1-26fb0def806e service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:25.314889 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0377498-6c12-44b6-8dc1-26fb0def806e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:25.329620 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0377498-6c12-44b6-8dc1-26fb0def806e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:25.331488 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0377498-6c12-44b6-8dc1-26fb0def806e service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:25.352078 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 733/1485] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:25 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 27 01:24:25.472927 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c9aa11-09d3-432d-89fe-033c9e5ec444 service cinder] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:25.475225 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-c0c9aa11-09d3-432d-89fe-033c9e5ec444 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:25.509823 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c9aa11-09d3-432d-89fe-033c9e5ec444 service cinder] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:25.526068 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-088ea120-f3fb-46c2-81bb-7ea4103a2874 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:25.527683 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088ea120-f3fb-46c2-81bb-7ea4103a2874 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:25.527963 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088ea120-f3fb-46c2-81bb-7ea4103a2874 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:25.528237 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088ea120-f3fb-46c2-81bb-7ea4103a2874 admin admin] PATH_INFO: `/v3/projects/5036cbc5dab947429bb10d03b1abd17e` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:25.533887 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-088ea120-f3fb-46c2-81bb-7ea4103a2874 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5036cbc5dab947429bb10d03b1abd17e)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:25.536428 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-088ea120-f3fb-46c2-81bb-7ea4103a2874 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:25.629603 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-088ea120-f3fb-46c2-81bb-7ea4103a2874 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5036cbc5dab947429bb10d03b1abd17e', 'request_id': 'req-088ea120-f3fb-46c2-81bb-7ea4103a2874'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:25.662214 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-088ea120-f3fb-46c2-81bb-7ea4103a2874 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5036cbc5dab947429bb10d03b1abd17e', 'request_id': 'req-088ea120-f3fb-46c2-81bb-7ea4103a2874'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:25.679075 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-088ea120-f3fb-46c2-81bb-7ea4103a2874 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5036cbc5dab947429bb10d03b1abd17e', 'request_id': 'req-088ea120-f3fb-46c2-81bb-7ea4103a2874'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:25.699431 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-088ea120-f3fb-46c2-81bb-7ea4103a2874 admin admin] The token cache is being invalidate because project 5036cbc5dab947429bb10d03b1abd17e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:24:25.699784 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-088ea120-f3fb-46c2-81bb-7ea4103a2874 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:25.700662 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-088ea120-f3fb-46c2-81bb-7ea4103a2874 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:25.711484 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 753/1486] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:24:25 2022] DELETE /identity/v3/projects/5036cbc5dab947429bb10d03b1abd17e => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:25.716310 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0377498-6c12-44b6-8dc1-26fb0def806e service nova] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:25.734695 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0377498-6c12-44b6-8dc1-26fb0def806e service nova] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:25.756908 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-088ea120-f3fb-46c2-81bb-7ea4103a2874 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:25.758519 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-088ea120-f3fb-46c2-81bb-7ea4103a2874 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:25.861656 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:25.923821 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-088ea120-f3fb-46c2-81bb-7ea4103a2874 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:25.932197 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0b4874-918f-4cd1-b81a-659c0d842ce3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:25.933219 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0b4874-918f-4cd1-b81a-659c0d842ce3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:25.940013 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a588f8-1339-40a8-939e-28a443464609 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:25.941275 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a588f8-1339-40a8-939e-28a443464609 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:25.941806 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a588f8-1339-40a8-939e-28a443464609 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:25.942271 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a588f8-1339-40a8-939e-28a443464609 admin admin] PATH_INFO: `/v3/users/048798141e664df8835afdd614a5d413` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:25.943798 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a588f8-1339-40a8-939e-28a443464609 admin admin] RBAC: Authorizing `identity:delete_user(user_id=048798141e664df8835afdd614a5d413)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:25.945912 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a588f8-1339-40a8-939e-28a443464609 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:25.954119 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0b4874-918f-4cd1-b81a-659c0d842ce3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:25.954653 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0b4874-918f-4cd1-b81a-659c0d842ce3 admin admin] PATH_INFO: `/v3/projects/a7e7557398ad411a92d3ec3f315b97bd` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:25.958649 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0b4874-918f-4cd1-b81a-659c0d842ce3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a7e7557398ad411a92d3ec3f315b97bd)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:25.960615 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0b4874-918f-4cd1-b81a-659c0d842ce3 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:26.043821 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-c5a588f8-1339-40a8-939e-28a443464609 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '048798141e664df8835afdd614a5d413'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:26.051818 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:26.054641 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-3b0b4874-918f-4cd1-b81a-659c0d842ce3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a7e7557398ad411a92d3ec3f315b97bd', 'request_id': 'req-3b0b4874-918f-4cd1-b81a-659c0d842ce3'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:26.062659 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-c5a588f8-1339-40a8-939e-28a443464609 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '048798141e664df8835afdd614a5d413'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:26.063770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-c5a588f8-1339-40a8-939e-28a443464609 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '048798141e664df8835afdd614a5d413'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:26.069336 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-3b0b4874-918f-4cd1-b81a-659c0d842ce3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a7e7557398ad411a92d3ec3f315b97bd', 'request_id': 'req-3b0b4874-918f-4cd1-b81a-659c0d842ce3'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:26.079168 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-3b0b4874-918f-4cd1-b81a-659c0d842ce3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a7e7557398ad411a92d3ec3f315b97bd', 'request_id': 'req-3b0b4874-918f-4cd1-b81a-659c0d842ce3'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:26.088126 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-c5a588f8-1339-40a8-939e-28a443464609 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '048798141e664df8835afdd614a5d413'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:26.095036 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-3b0b4874-918f-4cd1-b81a-659c0d842ce3 admin admin] The token cache is being invalidate because project a7e7557398ad411a92d3ec3f315b97bd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:24:26.095314 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-3b0b4874-918f-4cd1-b81a-659c0d842ce3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:26.095994 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-3b0b4874-918f-4cd1-b81a-659c0d842ce3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:26.098475 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 754/1487] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:24:25 2022] DELETE /identity/v3/projects/a7e7557398ad411a92d3ec3f315b97bd => generated 0 bytes in 342 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:26.102585 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-c5a588f8-1339-40a8-939e-28a443464609 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '048798141e664df8835afdd614a5d413'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:26.103358 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc59860-7aec-4c09-af85-ccf19066def7 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:26.103831 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc59860-7aec-4c09-af85-ccf19066def7 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:26.104177 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc59860-7aec-4c09-af85-ccf19066def7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:26.116700 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 734/1488] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:24:25 2022] DELETE /identity/v3/users/048798141e664df8835afdd614a5d413 => generated 0 bytes in 397 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:24:26.150061 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-7dc59860-7aec-4c09-af85-ccf19066def7 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:26.151801 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-7dc59860-7aec-4c09-af85-ccf19066def7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:26.258530 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 755/1489] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:24:26.259739 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf74339-a55b-4d5b-802d-77baaad07600 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:26.260025 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf74339-a55b-4d5b-802d-77baaad07600 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:26.260365 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf74339-a55b-4d5b-802d-77baaad07600 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:26.298999 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-6cf74339-a55b-4d5b-802d-77baaad07600 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:26.300875 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-6cf74339-a55b-4d5b-802d-77baaad07600 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:26.314514 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 735/1490] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 56 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:26.319724 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc59860-7aec-4c09-af85-ccf19066def7 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:26.330178 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc59860-7aec-4c09-af85-ccf19066def7 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:26.340112 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2150e430-ba6b-4302-a12f-9e900beddc34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:26.340885 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2150e430-ba6b-4302-a12f-9e900beddc34 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:26.341457 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2150e430-ba6b-4302-a12f-9e900beddc34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:26.341923 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2150e430-ba6b-4302-a12f-9e900beddc34 admin admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:26.343009 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2150e430-ba6b-4302-a12f-9e900beddc34 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:24:26.343506 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2150e430-ba6b-4302-a12f-9e900beddc34 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:26.347842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2150e430-ba6b-4302-a12f-9e900beddc34 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:26.366962 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 756/1491] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:24:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:24:26.368606 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf74339-a55b-4d5b-802d-77baaad07600 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:26.379452 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf74339-a55b-4d5b-802d-77baaad07600 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:26.386571 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aabf6647-5b81-4371-bc65-b72daf0e8a87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:26.387185 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabf6647-5b81-4371-bc65-b72daf0e8a87 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:26.391405 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabf6647-5b81-4371-bc65-b72daf0e8a87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:26.391686 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabf6647-5b81-4371-bc65-b72daf0e8a87 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:26.392346 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aabf6647-5b81-4371-bc65-b72daf0e8a87 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:26.395634 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aabf6647-5b81-4371-bc65-b72daf0e8a87 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:26.448540 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 736/1492] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:24:26 2022] POST /identity/v3/projects => generated 363 bytes in 80 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:24:26.452699 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2150e430-ba6b-4302-a12f-9e900beddc34 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:26.464850 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aabf6647-5b81-4371-bc65-b72daf0e8a87 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:26.468361 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-aabf6647-5b81-4371-bc65-b72daf0e8a87 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:26.483694 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2150e430-ba6b-4302-a12f-9e900beddc34 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:26.536980 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aabf6647-5b81-4371-bc65-b72daf0e8a87 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:26.541593 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1b6833-29d4-4217-9b9b-16328939b068 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:26.542224 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1b6833-29d4-4217-9b9b-16328939b068 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:26.542505 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1b6833-29d4-4217-9b9b-16328939b068 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:26.542777 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1b6833-29d4-4217-9b9b-16328939b068 admin admin] PATH_INFO: `/v3/projects/b2122c27e4ce4f149fed26e36cc0c360` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:26.543889 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf1b6833-29d4-4217-9b9b-16328939b068 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b2122c27e4ce4f149fed26e36cc0c360)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:26.545533 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e71090a0-f223-4784-9baf-79dabd406189 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:26.545836 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf1b6833-29d4-4217-9b9b-16328939b068 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:26.546140 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71090a0-f223-4784-9baf-79dabd406189 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:26.546423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71090a0-f223-4784-9baf-79dabd406189 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:26.546695 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71090a0-f223-4784-9baf-79dabd406189 admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:26.547318 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e71090a0-f223-4784-9baf-79dabd406189 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:26.549245 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e71090a0-f223-4784-9baf-79dabd406189 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:26.558002 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:26.575276 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 757/1493] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:24:26 2022] POST /identity/v3/users => generated 282 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:24:26.579011 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e71090a0-f223-4784-9baf-79dabd406189 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:26.589619 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e71090a0-f223-4784-9baf-79dabd406189 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:26.598597 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bf1b6833-29d4-4217-9b9b-16328939b068 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b2122c27e4ce4f149fed26e36cc0c360', 'request_id': 'req-bf1b6833-29d4-4217-9b9b-16328939b068'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:26.606109 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bf1b6833-29d4-4217-9b9b-16328939b068 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b2122c27e4ce4f149fed26e36cc0c360', 'request_id': 'req-bf1b6833-29d4-4217-9b9b-16328939b068'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:26.611593 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bf1b6833-29d4-4217-9b9b-16328939b068 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b2122c27e4ce4f149fed26e36cc0c360', 'request_id': 'req-bf1b6833-29d4-4217-9b9b-16328939b068'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:26.622794 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bf1b6833-29d4-4217-9b9b-16328939b068 admin admin] The token cache is being invalidate because project b2122c27e4ce4f149fed26e36cc0c360 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:24:26.623078 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bf1b6833-29d4-4217-9b9b-16328939b068 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:26.623757 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bf1b6833-29d4-4217-9b9b-16328939b068 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:26.625294 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 737/1494] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:24:26 2022] DELETE /identity/v3/projects/b2122c27e4ce4f149fed26e36cc0c360 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:26.631761 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55307f6d-a770-4c2b-8fdd-d266dea23404 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:26.632050 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55307f6d-a770-4c2b-8fdd-d266dea23404 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:26.632325 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55307f6d-a770-4c2b-8fdd-d266dea23404 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:26.649831 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9c5d4c-c58e-4d03-956d-b3444a92d297 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:26.650447 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9c5d4c-c58e-4d03-956d-b3444a92d297 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:26.650736 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9c5d4c-c58e-4d03-956d-b3444a92d297 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:26.651013 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9c5d4c-c58e-4d03-956d-b3444a92d297 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:26.651648 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd9c5d4c-c58e-4d03-956d-b3444a92d297 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:26.652784 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-55307f6d-a770-4c2b-8fdd-d266dea23404 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:26.653311 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd9c5d4c-c58e-4d03-956d-b3444a92d297 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:26.654041 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-55307f6d-a770-4c2b-8fdd-d266dea23404 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:26.663287 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 738/1495] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:24:26.665309 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 758/1496] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:26 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 27 01:24:26.671335 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55307f6d-a770-4c2b-8fdd-d266dea23404 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:26.671640 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6372831b-dc38-45e2-98aa-2389bfcd212b None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:26.671929 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6372831b-dc38-45e2-98aa-2389bfcd212b None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:26.672205 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6372831b-dc38-45e2-98aa-2389bfcd212b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:26.672686 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-55307f6d-a770-4c2b-8fdd-d266dea23404 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:26.685154 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55307f6d-a770-4c2b-8fdd-d266dea23404 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:26.689816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b48c02c-3dab-4a94-8dfa-2e4f800b1b89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:26.690937 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b48c02c-3dab-4a94-8dfa-2e4f800b1b89 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:26.691370 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b48c02c-3dab-4a94-8dfa-2e4f800b1b89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:26.691620 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b48c02c-3dab-4a94-8dfa-2e4f800b1b89 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:26.692257 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b48c02c-3dab-4a94-8dfa-2e4f800b1b89 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:26.692768 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-6372831b-dc38-45e2-98aa-2389bfcd212b None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:26.693880 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b48c02c-3dab-4a94-8dfa-2e4f800b1b89 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:26.693993 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-6372831b-dc38-45e2-98aa-2389bfcd212b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:26.702242 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 759/1497] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:26.708431 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 739/1498] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:26 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:26.708552 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6372831b-dc38-45e2-98aa-2389bfcd212b None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:26.709410 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b48c02c-3dab-4a94-8dfa-2e4f800b1b89 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:26.715996 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6372831b-dc38-45e2-98aa-2389bfcd212b None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:26.718947 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b48c02c-3dab-4a94-8dfa-2e4f800b1b89 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:26.720620 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4143874-9bd6-4e8c-9dc8-4d967efaa529 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:26.721240 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4143874-9bd6-4e8c-9dc8-4d967efaa529 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:26.721530 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4143874-9bd6-4e8c-9dc8-4d967efaa529 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:26.722396 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4143874-9bd6-4e8c-9dc8-4d967efaa529 admin admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:26.723154 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4143874-9bd6-4e8c-9dc8-4d967efaa529 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:24:26.723459 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4143874-9bd6-4e8c-9dc8-4d967efaa529 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:26.724570 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ef5c78-38c8-42fb-a4cc-64cdcc71ec5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:26.725092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4143874-9bd6-4e8c-9dc8-4d967efaa529 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:26.725362 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ef5c78-38c8-42fb-a4cc-64cdcc71ec5a admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:26.725649 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ef5c78-38c8-42fb-a4cc-64cdcc71ec5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:26.725925 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ef5c78-38c8-42fb-a4cc-64cdcc71ec5a admin admin] PATH_INFO: `/v3/projects/267a03e8dba3447baf87f211fc6fff93/users/2c18fc68ec844ccb8ad3c83a034c4485/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:26.736598 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 760/1499] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:24:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:24:26.737092 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=9,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:26.741540 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4143874-9bd6-4e8c-9dc8-4d967efaa529 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:26.744232 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32ef5c78-38c8-42fb-a4cc-64cdcc71ec5a admin admin] RBAC: Authorizing `identity:create_grant(project_id=267a03e8dba3447baf87f211fc6fff93, user_id=2c18fc68ec844ccb8ad3c83a034c4485, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:26.746974 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32ef5c78-38c8-42fb-a4cc-64cdcc71ec5a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:26.751842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4143874-9bd6-4e8c-9dc8-4d967efaa529 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:26.757983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-867217b7-a501-4047-af91-5473b78774e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:26.758700 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867217b7-a501-4047-af91-5473b78774e0 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:26.758852 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867217b7-a501-4047-af91-5473b78774e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:26.759125 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867217b7-a501-4047-af91-5473b78774e0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:26.759759 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-867217b7-a501-4047-af91-5473b78774e0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:26.761717 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-867217b7-a501-4047-af91-5473b78774e0 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:26.761908 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 740/1500] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:24:26 2022] PUT /identity/v3/projects/267a03e8dba3447baf87f211fc6fff93/users/2c18fc68ec844ccb8ad3c83a034c4485/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:24:26.775190 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583c9931-e8f7-466f-bb58-ff10dc565c00 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:26.775565 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583c9931-e8f7-466f-bb58-ff10dc565c00 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:26.775939 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583c9931-e8f7-466f-bb58-ff10dc565c00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:26.790181 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 761/1501] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:24:26 2022] POST /identity/v3/projects => generated 387 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:24:26.794305 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-867217b7-a501-4047-af91-5473b78774e0 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:26.807211 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-583c9931-e8f7-466f-bb58-ff10dc565c00 None None] MFA Rules not processed for user `2c18fc68ec844ccb8ad3c83a034c4485`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:26.808533 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-583c9931-e8f7-466f-bb58-ff10dc565c00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:26.861580 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-867217b7-a501-4047-af91-5473b78774e0 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:26.867333 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344f6c89-b371-415e-affc-37e14ac3987d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:26.867949 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344f6c89-b371-415e-affc-37e14ac3987d admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:26.868227 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344f6c89-b371-415e-affc-37e14ac3987d admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:26.868488 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344f6c89-b371-415e-affc-37e14ac3987d admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:26.869206 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-344f6c89-b371-415e-affc-37e14ac3987d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:26.871124 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-344f6c89-b371-415e-affc-37e14ac3987d admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:26.881705 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 741/1502] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:26 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:24:26.882727 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583c9931-e8f7-466f-bb58-ff10dc565c00 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:26.893095 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 762/1503] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:24:26 2022] POST /identity/v3/users => generated 294 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:24:26.897964 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344f6c89-b371-415e-affc-37e14ac3987d admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:26.898082 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583c9931-e8f7-466f-bb58-ff10dc565c00 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:26.902471 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01488fd6-42f1-4e4d-9011-df840c6c1cdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:26.903067 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01488fd6-42f1-4e4d-9011-df840c6c1cdc admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:26.903343 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01488fd6-42f1-4e4d-9011-df840c6c1cdc admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:26.903607 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01488fd6-42f1-4e4d-9011-df840c6c1cdc admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:26.904233 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01488fd6-42f1-4e4d-9011-df840c6c1cdc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:26.906162 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01488fd6-42f1-4e4d-9011-df840c6c1cdc admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:26.907870 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344f6c89-b371-415e-affc-37e14ac3987d admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:26.912883 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3488139-6f97-407f-b6be-9f52655bd45b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:26.913519 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3488139-6f97-407f-b6be-9f52655bd45b admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:26.913800 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3488139-6f97-407f-b6be-9f52655bd45b admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:26.914096 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3488139-6f97-407f-b6be-9f52655bd45b admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:26.914732 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3488139-6f97-407f-b6be-9f52655bd45b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:26.916313 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3488139-6f97-407f-b6be-9f52655bd45b admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:26.927500 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 763/1504] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:26 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 27 01:24:26.931979 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3488139-6f97-407f-b6be-9f52655bd45b admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:26.941791 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 742/1505] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:24:26 2022] POST /identity/v3/projects => generated 363 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:24:26.943067 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01488fd6-42f1-4e4d-9011-df840c6c1cdc admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:26.946720 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3488139-6f97-407f-b6be-9f52655bd45b admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:26.967007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:INSERT=7,keystone:SELECT=93 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:27.010398 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54e5fb4-63d7-4f18-a8fe-8922f9c21c54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:27.010990 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54e5fb4-63d7-4f18-a8fe-8922f9c21c54 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:27.011363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54e5fb4-63d7-4f18-a8fe-8922f9c21c54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:27.011528 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54e5fb4-63d7-4f18-a8fe-8922f9c21c54 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:27.012186 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54e5fb4-63d7-4f18-a8fe-8922f9c21c54 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:27.013803 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54e5fb4-63d7-4f18-a8fe-8922f9c21c54 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:27.028053 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 764/1506] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:26 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 27 01:24:27.028203 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01488fd6-42f1-4e4d-9011-df840c6c1cdc admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:27.029195 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54e5fb4-63d7-4f18-a8fe-8922f9c21c54 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:27.029998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a688aa0-ac4f-4b04-af3b-5339acab3cf2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:27.030615 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a688aa0-ac4f-4b04-af3b-5339acab3cf2 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:27.030890 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a688aa0-ac4f-4b04-af3b-5339acab3cf2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:27.031183 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a688aa0-ac4f-4b04-af3b-5339acab3cf2 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:27.031842 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a688aa0-ac4f-4b04-af3b-5339acab3cf2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:27.033798 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a688aa0-ac4f-4b04-af3b-5339acab3cf2 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:27.038740 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54e5fb4-63d7-4f18-a8fe-8922f9c21c54 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:27.043295 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9e975c-8c8e-412c-aa54-05f80fae67c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:27.043931 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9e975c-8c8e-412c-aa54-05f80fae67c3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:27.044226 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9e975c-8c8e-412c-aa54-05f80fae67c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:27.044510 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9e975c-8c8e-412c-aa54-05f80fae67c3 admin admin] PATH_INFO: `/v3/projects/4a09806fdd934b1abaddfe0009be5838/users/747af11c23f044439eef04c3e693da86/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:27.062829 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd9e975c-8c8e-412c-aa54-05f80fae67c3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4a09806fdd934b1abaddfe0009be5838, user_id=747af11c23f044439eef04c3e693da86, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:27.064610 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 743/1507] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:24:26 2022] POST /identity/v3/users => generated 288 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:24:27.067173 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd9e975c-8c8e-412c-aa54-05f80fae67c3 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:27.070475 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a688aa0-ac4f-4b04-af3b-5339acab3cf2 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:27.081134 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 765/1508] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:24:27 2022] PUT /identity/v3/projects/4a09806fdd934b1abaddfe0009be5838/users/747af11c23f044439eef04c3e693da86/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:24:27.085299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a688aa0-ac4f-4b04-af3b-5339acab3cf2 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:27.093822 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618128c1-563d-41bf-91ff-a401e1a3e384 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:27.094104 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618128c1-563d-41bf-91ff-a401e1a3e384 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:27.094385 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618128c1-563d-41bf-91ff-a401e1a3e384 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:27.122592 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-618128c1-563d-41bf-91ff-a401e1a3e384 None None] MFA Rules not processed for user `747af11c23f044439eef04c3e693da86`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:27.123791 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-618128c1-563d-41bf-91ff-a401e1a3e384 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:27.129634 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:27.144429 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbb0754-0535-4aa2-957f-b00355c68c34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:27.145279 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbb0754-0535-4aa2-957f-b00355c68c34 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:27.145556 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbb0754-0535-4aa2-957f-b00355c68c34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:27.145826 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbb0754-0535-4aa2-957f-b00355c68c34 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:27.146492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcbb0754-0535-4aa2-957f-b00355c68c34 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:27.148080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcbb0754-0535-4aa2-957f-b00355c68c34 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:27.159788 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 744/1509] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:27 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:27.164728 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbb0754-0535-4aa2-957f-b00355c68c34 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:27.174822 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbb0754-0535-4aa2-957f-b00355c68c34 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:27.179391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0684ddfc-8ca0-47c0-a311-f1b673026d07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:27.180020 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0684ddfc-8ca0-47c0-a311-f1b673026d07 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:27.180300 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0684ddfc-8ca0-47c0-a311-f1b673026d07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:27.180932 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0684ddfc-8ca0-47c0-a311-f1b673026d07 admin admin] PATH_INFO: `/v3/projects/e991555393b147c594f35fabd0c8ba4f/users/b106f043ba8649b9a36941d09a3e1ec6/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:27.196077 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 766/1510] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:27 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:24:27.198371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0684ddfc-8ca0-47c0-a311-f1b673026d07 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e991555393b147c594f35fabd0c8ba4f, user_id=b106f043ba8649b9a36941d09a3e1ec6, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:27.200239 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618128c1-563d-41bf-91ff-a401e1a3e384 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:27.201342 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0684ddfc-8ca0-47c0-a311-f1b673026d07 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:27.213409 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618128c1-563d-41bf-91ff-a401e1a3e384 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:27.217650 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 745/1511] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:24:27 2022] PUT /identity/v3/projects/e991555393b147c594f35fabd0c8ba4f/users/b106f043ba8649b9a36941d09a3e1ec6/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:24:27.220320 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0684ddfc-8ca0-47c0-a311-f1b673026d07 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:27.285770 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a605fb97-eb14-4802-9af1-43548573d1c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:27.286383 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a605fb97-eb14-4802-9af1-43548573d1c8 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:27.286663 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a605fb97-eb14-4802-9af1-43548573d1c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:27.286932 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a605fb97-eb14-4802-9af1-43548573d1c8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:27.287563 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a605fb97-eb14-4802-9af1-43548573d1c8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:27.289556 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a605fb97-eb14-4802-9af1-43548573d1c8 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:27.298820 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0684ddfc-8ca0-47c0-a311-f1b673026d07 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:27.307928 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a4a443-d9e1-41e1-8ce1-b141ed11d920 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:27.308672 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a4a443-d9e1-41e1-8ce1-b141ed11d920 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:27.308960 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a4a443-d9e1-41e1-8ce1-b141ed11d920 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:27.309359 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a4a443-d9e1-41e1-8ce1-b141ed11d920 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:27.310080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a4a443-d9e1-41e1-8ce1-b141ed11d920 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:27.311977 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a4a443-d9e1-41e1-8ce1-b141ed11d920 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:27.322165 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 767/1512] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:24:27 2022] POST /identity/v3/projects => generated 387 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:24:27.325761 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 746/1513] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:27 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:27.328937 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a605fb97-eb14-4802-9af1-43548573d1c8 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:27.329552 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a4a443-d9e1-41e1-8ce1-b141ed11d920 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:27.340662 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a4a443-d9e1-41e1-8ce1-b141ed11d920 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:27.342765 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:27.407419 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e02d2d3a-11ee-45be-a5c4-994618915546 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:27.408894 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02d2d3a-11ee-45be-a5c4-994618915546 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:27.408894 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02d2d3a-11ee-45be-a5c4-994618915546 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:27.408894 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02d2d3a-11ee-45be-a5c4-994618915546 admin admin] PATH_INFO: `/v3/projects/e991555393b147c594f35fabd0c8ba4f/users/b106f043ba8649b9a36941d09a3e1ec6/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:27.412029 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e02d2d3a-11ee-45be-a5c4-994618915546 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e991555393b147c594f35fabd0c8ba4f, user_id=b106f043ba8649b9a36941d09a3e1ec6, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:27.413915 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a605fb97-eb14-4802-9af1-43548573d1c8 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:27.415232 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e02d2d3a-11ee-45be-a5c4-994618915546 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:27.418921 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac5dea3-68e4-423a-b109-bd2fa761bf6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:27.419604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac5dea3-68e4-423a-b109-bd2fa761bf6c admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:27.419948 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac5dea3-68e4-423a-b109-bd2fa761bf6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:27.420316 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac5dea3-68e4-423a-b109-bd2fa761bf6c admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:27.420997 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac5dea3-68e4-423a-b109-bd2fa761bf6c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:27.423022 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac5dea3-68e4-423a-b109-bd2fa761bf6c admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:27.447959 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 747/1514] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:24:27 2022] PUT /identity/v3/projects/e991555393b147c594f35fabd0c8ba4f/users/b106f043ba8649b9a36941d09a3e1ec6/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:24:27.462207 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 768/1515] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:24:27 2022] POST /identity/v3/users => generated 300 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:24:27.465414 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897dbc34-4ee6-4565-9215-456a76dc7e44 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:27.465613 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897dbc34-4ee6-4565-9215-456a76dc7e44 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:27.465882 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897dbc34-4ee6-4565-9215-456a76dc7e44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:27.466455 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac5dea3-68e4-423a-b109-bd2fa761bf6c admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:27.481849 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac5dea3-68e4-423a-b109-bd2fa761bf6c admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:27.512654 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-897dbc34-4ee6-4565-9215-456a76dc7e44 None None] MFA Rules not processed for user `b106f043ba8649b9a36941d09a3e1ec6`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:27.514179 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-897dbc34-4ee6-4565-9215-456a76dc7e44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:27.571868 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1be7da4-1228-445c-9c6f-7593a76f5dc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:27.572747 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1be7da4-1228-445c-9c6f-7593a76f5dc6 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:27.573203 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1be7da4-1228-445c-9c6f-7593a76f5dc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:27.573647 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1be7da4-1228-445c-9c6f-7593a76f5dc6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:27.575170 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1be7da4-1228-445c-9c6f-7593a76f5dc6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:27.576568 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:27.577166 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1be7da4-1228-445c-9c6f-7593a76f5dc6 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:27.591233 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 769/1516] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:27 2022] GET /identity/v3/roles => generated 1420 bytes in 122 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:27.593218 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1be7da4-1228-445c-9c6f-7593a76f5dc6 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:27.614299 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1be7da4-1228-445c-9c6f-7593a76f5dc6 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:27.619885 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 748/1517] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:27 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:24:27.624324 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf7d35f-c463-436e-be2e-4456d18c5a5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:27.624999 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf7d35f-c463-436e-be2e-4456d18c5a5c admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:27.625305 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf7d35f-c463-436e-be2e-4456d18c5a5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:27.625611 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf7d35f-c463-436e-be2e-4456d18c5a5c admin admin] PATH_INFO: `/v3/projects/59d427277fc14351afba219c9e6eb1f5/users/5792ae31ece0486aadb35b528463d355/roles/414286927fdd44688107ff5e10235acf` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:27.631743 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897dbc34-4ee6-4565-9215-456a76dc7e44 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:27.633129 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-897dbc34-4ee6-4565-9215-456a76dc7e44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:27.642390 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf7d35f-c463-436e-be2e-4456d18c5a5c admin admin] RBAC: Authorizing `identity:create_grant(project_id=59d427277fc14351afba219c9e6eb1f5, user_id=5792ae31ece0486aadb35b528463d355, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:27.646851 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf7d35f-c463-436e-be2e-4456d18c5a5c admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:27.681309 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 770/1518] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:24:27 2022] PUT /identity/v3/projects/59d427277fc14351afba219c9e6eb1f5/users/5792ae31ece0486aadb35b528463d355/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 86 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:27.683460 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf7d35f-c463-436e-be2e-4456d18c5a5c admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:27.704622 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897dbc34-4ee6-4565-9215-456a76dc7e44 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:27.764816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-024f56d1-7feb-4783-bb71-8380b519c44f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:27.765527 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024f56d1-7feb-4783-bb71-8380b519c44f service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:27.765874 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024f56d1-7feb-4783-bb71-8380b519c44f service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:27.766135 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024f56d1-7feb-4783-bb71-8380b519c44f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:27.771551 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf7d35f-c463-436e-be2e-4456d18c5a5c admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:27.776897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9237be-1699-481d-85f1-011cbf4a8eb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:27.777560 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9237be-1699-481d-85f1-011cbf4a8eb5 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:27.777835 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9237be-1699-481d-85f1-011cbf4a8eb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:27.778105 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9237be-1699-481d-85f1-011cbf4a8eb5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:27.778799 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab9237be-1699-481d-85f1-011cbf4a8eb5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:27.780424 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab9237be-1699-481d-85f1-011cbf4a8eb5 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:27.819028 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 771/1519] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:27 2022] GET /identity/v3/roles => generated 1420 bytes in 132 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:27.820821 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9237be-1699-481d-85f1-011cbf4a8eb5 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:27.833946 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9237be-1699-481d-85f1-011cbf4a8eb5 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:27.838891 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ee4249-2249-4d97-92dd-17c45ed60727 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:27.839537 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ee4249-2249-4d97-92dd-17c45ed60727 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:27.839834 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ee4249-2249-4d97-92dd-17c45ed60727 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:27.840207 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ee4249-2249-4d97-92dd-17c45ed60727 admin admin] PATH_INFO: `/v3/projects/59d427277fc14351afba219c9e6eb1f5/users/5792ae31ece0486aadb35b528463d355/roles/414286927fdd44688107ff5e10235acf` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:27.842198 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ee4249-2249-4d97-92dd-17c45ed60727 admin admin] RBAC: Authorizing `identity:create_grant(project_id=59d427277fc14351afba219c9e6eb1f5, user_id=5792ae31ece0486aadb35b528463d355, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:27.844826 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ee4249-2249-4d97-92dd-17c45ed60727 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:27.848671 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:27.863390 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 772/1520] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:24:27 2022] PUT /identity/v3/projects/59d427277fc14351afba219c9e6eb1f5/users/5792ae31ece0486aadb35b528463d355/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:27.874264 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaffa2a8-f509-421d-8d95-b0a48d2450af None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:27.874668 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaffa2a8-f509-421d-8d95-b0a48d2450af None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:27.875101 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaffa2a8-f509-421d-8d95-b0a48d2450af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:27.904914 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-024f56d1-7feb-4783-bb71-8380b519c44f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:27.910368 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-024f56d1-7feb-4783-bb71-8380b519c44f service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:27.935426 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 749/1521] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:27 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 27 01:24:27.958085 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-aaffa2a8-f509-421d-8d95-b0a48d2450af None None] MFA Rules not processed for user `5792ae31ece0486aadb35b528463d355`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:27.959784 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-aaffa2a8-f509-421d-8d95-b0a48d2450af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:28.078626 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-024f56d1-7feb-4783-bb71-8380b519c44f service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:28.087978 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-024f56d1-7feb-4783-bb71-8380b519c44f service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:28.109576 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 773/1522] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:27 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 234 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:28.119232 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaffa2a8-f509-421d-8d95-b0a48d2450af None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:28.120632 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-aaffa2a8-f509-421d-8d95-b0a48d2450af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:28.137477 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:28.156231 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45ef115-06b6-47df-bdc0-8f732d22a7dd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:28.156889 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45ef115-06b6-47df-bdc0-8f732d22a7dd service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:28.157201 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45ef115-06b6-47df-bdc0-8f732d22a7dd service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:28.157614 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45ef115-06b6-47df-bdc0-8f732d22a7dd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:28.193989 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaffa2a8-f509-421d-8d95-b0a48d2450af None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:28.198080 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591d899d-1a94-4d23-9262-ae5a37e69bab service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:28.198671 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591d899d-1a94-4d23-9262-ae5a37e69bab service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:28.198950 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591d899d-1a94-4d23-9262-ae5a37e69bab service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:28.200534 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591d899d-1a94-4d23-9262-ae5a37e69bab service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:28.210645 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-591d899d-1a94-4d23-9262-ae5a37e69bab service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:28.212854 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-591d899d-1a94-4d23-9262-ae5a37e69bab service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:28.229752 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b45ef115-06b6-47df-bdc0-8f732d22a7dd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:28.230858 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 774/1523] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:28 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 27 01:24:28.244593 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b45ef115-06b6-47df-bdc0-8f732d22a7dd service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:28.257886 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 750/1524] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:28 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 180 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 27 01:24:29.347914 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591d899d-1a94-4d23-9262-ae5a37e69bab service cinder] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:29.430213 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45ef115-06b6-47df-bdc0-8f732d22a7dd service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:29.446609 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45ef115-06b6-47df-bdc0-8f732d22a7dd service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:29.456477 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0c4a96-c7e6-419d-8369-37b6aba4b2c9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:29.457146 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0c4a96-c7e6-419d-8369-37b6aba4b2c9 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:29.457426 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0c4a96-c7e6-419d-8369-37b6aba4b2c9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:29.457704 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0c4a96-c7e6-419d-8369-37b6aba4b2c9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:29.462999 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-8f0c4a96-c7e6-419d-8369-37b6aba4b2c9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:29.480369 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591d899d-1a94-4d23-9262-ae5a37e69bab service cinder] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:29.496004 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1380cf-d64c-474b-a5b8-6212f76c7b20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:29.496765 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1380cf-d64c-474b-a5b8-6212f76c7b20 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:29.497155 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1380cf-d64c-474b-a5b8-6212f76c7b20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:29.497495 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1380cf-d64c-474b-a5b8-6212f76c7b20 admin admin] PATH_INFO: `/v3/users/747af11c23f044439eef04c3e693da86` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:29.499438 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1380cf-d64c-474b-a5b8-6212f76c7b20 admin admin] RBAC: Authorizing `identity:delete_user(user_id=747af11c23f044439eef04c3e693da86)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:29.501316 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1380cf-d64c-474b-a5b8-6212f76c7b20 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:29.539797 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:INSERT=1,keystone:SELECT=95,keystone:DELETE=4 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:29.599553 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-4e1380cf-d64c-474b-a5b8-6212f76c7b20 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '747af11c23f044439eef04c3e693da86'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:29.612722 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-4e1380cf-d64c-474b-a5b8-6212f76c7b20 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '747af11c23f044439eef04c3e693da86'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:29.619554 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f0c4a96-c7e6-419d-8369-37b6aba4b2c9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:29.625395 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f0c4a96-c7e6-419d-8369-37b6aba4b2c9 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:29.645794 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-4e1380cf-d64c-474b-a5b8-6212f76c7b20 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '747af11c23f044439eef04c3e693da86'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:29.646897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-4e1380cf-d64c-474b-a5b8-6212f76c7b20 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '747af11c23f044439eef04c3e693da86'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:29.647939 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 751/1525] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:29 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 218 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 27 01:24:29.648371 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-4e1380cf-d64c-474b-a5b8-6212f76c7b20 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '747af11c23f044439eef04c3e693da86'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:29.677125 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 775/1526] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:24:29 2022] DELETE /identity/v3/users/747af11c23f044439eef04c3e693da86 => generated 0 bytes in 329 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:29.681302 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b39b97f-c63d-4355-976f-eca10222d470 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:29.682003 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b39b97f-c63d-4355-976f-eca10222d470 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:29.682384 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b39b97f-c63d-4355-976f-eca10222d470 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:29.723368 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-3b39b97f-c63d-4355-976f-eca10222d470 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:29.725028 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-3b39b97f-c63d-4355-976f-eca10222d470 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:29.840913 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 752/1527] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:29.860962 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1380cf-d64c-474b-a5b8-6212f76c7b20 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:29.863340 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-4e1380cf-d64c-474b-a5b8-6212f76c7b20 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:30.052538 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1380cf-d64c-474b-a5b8-6212f76c7b20 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:30.057493 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5b7e53-a71b-48e0-b880-e562a3233c89 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:30.058284 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5b7e53-a71b-48e0-b880-e562a3233c89 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:30.058622 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5b7e53-a71b-48e0-b880-e562a3233c89 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:30.059053 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5b7e53-a71b-48e0-b880-e562a3233c89 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:30.092175 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c5b7e53-a71b-48e0-b880-e562a3233c89 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:30.095077 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c5b7e53-a71b-48e0-b880-e562a3233c89 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:30.121720 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 776/1528] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 261 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:24:30.783606 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b39b97f-c63d-4355-976f-eca10222d470 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:30.786644 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-3b39b97f-c63d-4355-976f-eca10222d470 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:30.803235 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b39b97f-c63d-4355-976f-eca10222d470 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:30.808954 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fcb450-1633-4072-811c-ccd20b8012c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:30.809785 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fcb450-1633-4072-811c-ccd20b8012c1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:30.810169 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fcb450-1633-4072-811c-ccd20b8012c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:30.810589 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fcb450-1633-4072-811c-ccd20b8012c1 admin admin] PATH_INFO: `/v3/projects/4a09806fdd934b1abaddfe0009be5838` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:30.811785 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5fcb450-1633-4072-811c-ccd20b8012c1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4a09806fdd934b1abaddfe0009be5838)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:30.814344 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5fcb450-1633-4072-811c-ccd20b8012c1 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:30.855905 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:30.876713 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-e5fcb450-1633-4072-811c-ccd20b8012c1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4a09806fdd934b1abaddfe0009be5838', 'request_id': 'req-e5fcb450-1633-4072-811c-ccd20b8012c1'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:30.886377 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-e5fcb450-1633-4072-811c-ccd20b8012c1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4a09806fdd934b1abaddfe0009be5838', 'request_id': 'req-e5fcb450-1633-4072-811c-ccd20b8012c1'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:30.892550 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-e5fcb450-1633-4072-811c-ccd20b8012c1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4a09806fdd934b1abaddfe0009be5838', 'request_id': 'req-e5fcb450-1633-4072-811c-ccd20b8012c1'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:30.897114 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5b7e53-a71b-48e0-b880-e562a3233c89 service neutron] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:30.899080 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-1c5b7e53-a71b-48e0-b880-e562a3233c89 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:30.906641 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-e5fcb450-1633-4072-811c-ccd20b8012c1 admin admin] The token cache is being invalidate because project 4a09806fdd934b1abaddfe0009be5838 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:24:30.906940 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-e5fcb450-1633-4072-811c-ccd20b8012c1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:30.907922 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-e5fcb450-1633-4072-811c-ccd20b8012c1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:30.909771 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 753/1529] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:24:30 2022] DELETE /identity/v3/projects/4a09806fdd934b1abaddfe0009be5838 => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:30.913675 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fcb450-1633-4072-811c-ccd20b8012c1 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:30.915085 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e5fcb450-1633-4072-811c-ccd20b8012c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:30.967358 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5b7e53-a71b-48e0-b880-e562a3233c89 service neutron] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:30.997595 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fcb450-1633-4072-811c-ccd20b8012c1 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:31.004560 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895368ae-4a90-4cc0-86a0-1bb3f2da507c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:31.005278 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895368ae-4a90-4cc0-86a0-1bb3f2da507c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:31.006173 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895368ae-4a90-4cc0-86a0-1bb3f2da507c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:31.006577 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895368ae-4a90-4cc0-86a0-1bb3f2da507c admin admin] PATH_INFO: `/v3/users/5792ae31ece0486aadb35b528463d355` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:31.007949 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-895368ae-4a90-4cc0-86a0-1bb3f2da507c admin admin] RBAC: Authorizing `identity:delete_user(user_id=5792ae31ece0486aadb35b528463d355)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:31.010297 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-895368ae-4a90-4cc0-86a0-1bb3f2da507c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:31.031554 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3ee333-acef-4aaf-a9a2-16165c6c96c5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:31.032405 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3ee333-acef-4aaf-a9a2-16165c6c96c5 service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:31.032832 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3ee333-acef-4aaf-a9a2-16165c6c96c5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:31.033279 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3ee333-acef-4aaf-a9a2-16165c6c96c5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:31.035749 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-8a3ee333-acef-4aaf-a9a2-16165c6c96c5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:31.057773 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=93,keystone:DELETE=6,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:31.075662 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-895368ae-4a90-4cc0-86a0-1bb3f2da507c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5792ae31ece0486aadb35b528463d355'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:31.086359 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-895368ae-4a90-4cc0-86a0-1bb3f2da507c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5792ae31ece0486aadb35b528463d355'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:31.087174 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-895368ae-4a90-4cc0-86a0-1bb3f2da507c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5792ae31ece0486aadb35b528463d355'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:31.104682 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-895368ae-4a90-4cc0-86a0-1bb3f2da507c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5792ae31ece0486aadb35b528463d355'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:31.114077 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-895368ae-4a90-4cc0-86a0-1bb3f2da507c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5792ae31ece0486aadb35b528463d355'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:31.120701 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 754/1530] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:24:30 2022] DELETE /identity/v3/users/5792ae31ece0486aadb35b528463d355 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:24:31.132505 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a3ee333-acef-4aaf-a9a2-16165c6c96c5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:31.133988 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a3ee333-acef-4aaf-a9a2-16165c6c96c5 service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:31.144004 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 777/1531] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:30 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 248 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 27 01:24:31.583190 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895368ae-4a90-4cc0-86a0-1bb3f2da507c admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:31.596380 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-895368ae-4a90-4cc0-86a0-1bb3f2da507c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:31.664453 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:31.695176 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895368ae-4a90-4cc0-86a0-1bb3f2da507c admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:31.703610 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2848396d-7230-40e8-b069-72e4b7b1164e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:31.704474 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2848396d-7230-40e8-b069-72e4b7b1164e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:31.704908 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2848396d-7230-40e8-b069-72e4b7b1164e admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:31.705216 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2848396d-7230-40e8-b069-72e4b7b1164e admin admin] PATH_INFO: `/v3/projects/59d427277fc14351afba219c9e6eb1f5` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:31.707937 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2848396d-7230-40e8-b069-72e4b7b1164e admin admin] RBAC: Authorizing `identity:delete_project(project_id=59d427277fc14351afba219c9e6eb1f5)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:31.712571 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2848396d-7230-40e8-b069-72e4b7b1164e admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:31.728552 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3ee333-acef-4aaf-a9a2-16165c6c96c5 service cinder] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:31.730069 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-8a3ee333-acef-4aaf-a9a2-16165c6c96c5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:31.744860 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3ee333-acef-4aaf-a9a2-16165c6c96c5 service cinder] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:31.752378 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdff40ae-4816-45f0-9546-563f62a89139 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:31.754154 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdff40ae-4816-45f0-9546-563f62a89139 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:31.754434 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdff40ae-4816-45f0-9546-563f62a89139 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:31.754708 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdff40ae-4816-45f0-9546-563f62a89139 admin admin] PATH_INFO: `/v3/users/2c18fc68ec844ccb8ad3c83a034c4485` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:31.757043 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdff40ae-4816-45f0-9546-563f62a89139 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2c18fc68ec844ccb8ad3c83a034c4485)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:31.758969 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdff40ae-4816-45f0-9546-563f62a89139 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:31.773190 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2848396d-7230-40e8-b069-72e4b7b1164e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '59d427277fc14351afba219c9e6eb1f5', 'request_id': 'req-2848396d-7230-40e8-b069-72e4b7b1164e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:31.783268 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2848396d-7230-40e8-b069-72e4b7b1164e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '59d427277fc14351afba219c9e6eb1f5', 'request_id': 'req-2848396d-7230-40e8-b069-72e4b7b1164e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:31.792854 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2848396d-7230-40e8-b069-72e4b7b1164e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '59d427277fc14351afba219c9e6eb1f5', 'request_id': 'req-2848396d-7230-40e8-b069-72e4b7b1164e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:31.809624 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2848396d-7230-40e8-b069-72e4b7b1164e admin admin] The token cache is being invalidate because project 59d427277fc14351afba219c9e6eb1f5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:24:31.809918 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2848396d-7230-40e8-b069-72e4b7b1164e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:31.813771 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2848396d-7230-40e8-b069-72e4b7b1164e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:31.818178 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 755/1532] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:24:31 2022] DELETE /identity/v3/projects/59d427277fc14351afba219c9e6eb1f5 => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:31.823072 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82879088-8aa2-41c7-9506-4e92d350eafd None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:31.823349 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82879088-8aa2-41c7-9506-4e92d350eafd None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:31.823623 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82879088-8aa2-41c7-9506-4e92d350eafd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:31.825322 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-bdff40ae-4816-45f0-9546-563f62a89139 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2c18fc68ec844ccb8ad3c83a034c4485'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:31.837428 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-bdff40ae-4816-45f0-9546-563f62a89139 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2c18fc68ec844ccb8ad3c83a034c4485'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:31.857622 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-82879088-8aa2-41c7-9506-4e92d350eafd None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:31.861570 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-82879088-8aa2-41c7-9506-4e92d350eafd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:31.873177 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-bdff40ae-4816-45f0-9546-563f62a89139 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2c18fc68ec844ccb8ad3c83a034c4485'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:31.874066 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-bdff40ae-4816-45f0-9546-563f62a89139 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2c18fc68ec844ccb8ad3c83a034c4485'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:31.874829 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-bdff40ae-4816-45f0-9546-563f62a89139 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2c18fc68ec844ccb8ad3c83a034c4485'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:31.892691 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 778/1533] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:24:31 2022] DELETE /identity/v3/users/2c18fc68ec844ccb8ad3c83a034c4485 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:24:31.898058 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d0d97b-d658-47b3-b1cc-d9c46fa9084d None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:31.898697 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d0d97b-d658-47b3-b1cc-d9c46fa9084d None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:31.898989 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d0d97b-d658-47b3-b1cc-d9c46fa9084d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:31.927284 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-59d0d97b-d658-47b3-b1cc-d9c46fa9084d None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:31.929554 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-59d0d97b-d658-47b3-b1cc-d9c46fa9084d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:31.975390 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 756/1534] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:24:31.984572 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3eaf99-ac5e-4c86-9af8-9de7bbcea191 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:31.984930 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3eaf99-ac5e-4c86-9af8-9de7bbcea191 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:31.985440 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3eaf99-ac5e-4c86-9af8-9de7bbcea191 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:32.010861 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 779/1535] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:32.021299 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59d0d97b-d658-47b3-b1cc-d9c46fa9084d None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:32.023114 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-59d0d97b-d658-47b3-b1cc-d9c46fa9084d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:32.030154 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-6a3eaf99-ac5e-4c86-9af8-9de7bbcea191 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:32.031975 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-6a3eaf99-ac5e-4c86-9af8-9de7bbcea191 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:32.041160 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 757/1536] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:32.045322 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3eaf99-ac5e-4c86-9af8-9de7bbcea191 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:32.051380 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:32.055323 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3eaf99-ac5e-4c86-9af8-9de7bbcea191 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:32.060672 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc08a06-26b3-4754-874e-4f46982a79e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:32.061321 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc08a06-26b3-4754-874e-4f46982a79e8 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:32.061596 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc08a06-26b3-4754-874e-4f46982a79e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:32.061869 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc08a06-26b3-4754-874e-4f46982a79e8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:32.062519 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc08a06-26b3-4754-874e-4f46982a79e8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:24:32.062820 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc08a06-26b3-4754-874e-4f46982a79e8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:32.064407 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc08a06-26b3-4754-874e-4f46982a79e8 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:32.081183 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 758/1537] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:24:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:24:32.082937 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc08a06-26b3-4754-874e-4f46982a79e8 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:32.094426 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59d0d97b-d658-47b3-b1cc-d9c46fa9084d None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:32.094582 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc08a06-26b3-4754-874e-4f46982a79e8 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:32.104424 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96c502a-b12f-426a-ac5b-45267d04652e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:32.105486 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96c502a-b12f-426a-ac5b-45267d04652e service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:32.106066 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96c502a-b12f-426a-ac5b-45267d04652e service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:32.106753 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96c502a-b12f-426a-ac5b-45267d04652e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:32.108102 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455e38fd-93de-4148-84a4-a00ff94c5575 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:32.108808 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455e38fd-93de-4148-84a4-a00ff94c5575 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:32.109373 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455e38fd-93de-4148-84a4-a00ff94c5575 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:32.109675 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455e38fd-93de-4148-84a4-a00ff94c5575 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:32.110450 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-455e38fd-93de-4148-84a4-a00ff94c5575 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:32.113127 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-455e38fd-93de-4148-84a4-a00ff94c5575 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:32.133600 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c96c502a-b12f-426a-ac5b-45267d04652e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:32.135078 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c96c502a-b12f-426a-ac5b-45267d04652e service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:32.148561 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 780/1538] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 27 01:24:32.157087 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 759/1539] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:24:32 2022] POST /identity/v3/projects => generated 373 bytes in 72 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:24:32.158209 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96c502a-b12f-426a-ac5b-45267d04652e service neutron] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:32.169861 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96c502a-b12f-426a-ac5b-45267d04652e service neutron] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:32.262291 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a2bc5a-be89-4805-91cd-511c0e5dc785 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:32.263362 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a2bc5a-be89-4805-91cd-511c0e5dc785 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:32.263627 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a2bc5a-be89-4805-91cd-511c0e5dc785 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:32.264181 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a2bc5a-be89-4805-91cd-511c0e5dc785 admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:32.264948 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a2bc5a-be89-4805-91cd-511c0e5dc785 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:32.267220 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a2bc5a-be89-4805-91cd-511c0e5dc785 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:32.321348 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 781/1540] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:24:32 2022] POST /identity/v3/users => generated 287 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:24:32.323915 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455e38fd-93de-4148-84a4-a00ff94c5575 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:32.339388 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455e38fd-93de-4148-84a4-a00ff94c5575 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:32.352892 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a2bc5a-be89-4805-91cd-511c0e5dc785 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:32.354839 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-f8a2bc5a-be89-4805-91cd-511c0e5dc785 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:32.362058 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d59cb7-3cc6-4f59-be8e-04e96a9c5caa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:32.363224 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d59cb7-3cc6-4f59-be8e-04e96a9c5caa admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:32.363491 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d59cb7-3cc6-4f59-be8e-04e96a9c5caa admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:32.363838 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d59cb7-3cc6-4f59-be8e-04e96a9c5caa admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:32.364412 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d59cb7-3cc6-4f59-be8e-04e96a9c5caa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:32.366046 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d59cb7-3cc6-4f59-be8e-04e96a9c5caa admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:32.377662 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a2bc5a-be89-4805-91cd-511c0e5dc785 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:32.387558 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a164a6e-a7a2-48a6-a21c-0f39b2735c1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:32.388285 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a164a6e-a7a2-48a6-a21c-0f39b2735c1e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:32.388570 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a164a6e-a7a2-48a6-a21c-0f39b2735c1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:32.388843 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a164a6e-a7a2-48a6-a21c-0f39b2735c1e admin admin] PATH_INFO: `/v3/users/fc808f90939541c7a0492c80cca2f8d7` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:32.389686 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 760/1541] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:32 2022] GET /identity/v3/roles => generated 1420 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:32.390442 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a164a6e-a7a2-48a6-a21c-0f39b2735c1e admin admin] RBAC: Authorizing `identity:delete_user(user_id=fc808f90939541c7a0492c80cca2f8d7)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:32.392396 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a164a6e-a7a2-48a6-a21c-0f39b2735c1e admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:32.396047 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d59cb7-3cc6-4f59-be8e-04e96a9c5caa admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:32.418351 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d59cb7-3cc6-4f59-be8e-04e96a9c5caa admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:32.427141 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2eb6121-3e6f-4767-9133-99ada27b734a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:32.427833 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2eb6121-3e6f-4767-9133-99ada27b734a admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:32.428158 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2eb6121-3e6f-4767-9133-99ada27b734a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:32.428375 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2eb6121-3e6f-4767-9133-99ada27b734a admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:32.429297 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2eb6121-3e6f-4767-9133-99ada27b734a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:32.431313 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2eb6121-3e6f-4767-9133-99ada27b734a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:32.453252 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 761/1542] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:32 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:32.458402 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2eb6121-3e6f-4767-9133-99ada27b734a admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:32.469839 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:32.486925 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2eb6121-3e6f-4767-9133-99ada27b734a admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:32.493518 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-4a164a6e-a7a2-48a6-a21c-0f39b2735c1e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fc808f90939541c7a0492c80cca2f8d7'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:32.498751 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940efb1e-514e-461a-a1aa-767a3485c91a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:32.499621 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940efb1e-514e-461a-a1aa-767a3485c91a admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:32.500007 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940efb1e-514e-461a-a1aa-767a3485c91a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:32.500313 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940efb1e-514e-461a-a1aa-767a3485c91a admin admin] PATH_INFO: `/v3/projects/c3e3b33974604b8f8cdb5f4849917a4b/users/248eb53661d24b3380de06b7f6b10dd8/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:32.506330 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-4a164a6e-a7a2-48a6-a21c-0f39b2735c1e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fc808f90939541c7a0492c80cca2f8d7'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:32.532820 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-4a164a6e-a7a2-48a6-a21c-0f39b2735c1e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fc808f90939541c7a0492c80cca2f8d7'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:32.533944 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-4a164a6e-a7a2-48a6-a21c-0f39b2735c1e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fc808f90939541c7a0492c80cca2f8d7'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:32.535300 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940efb1e-514e-461a-a1aa-767a3485c91a admin admin] RBAC: Authorizing `identity:create_grant(project_id=c3e3b33974604b8f8cdb5f4849917a4b, user_id=248eb53661d24b3380de06b7f6b10dd8, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:32.538023 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940efb1e-514e-461a-a1aa-767a3485c91a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:32.540839 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-4a164a6e-a7a2-48a6-a21c-0f39b2735c1e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fc808f90939541c7a0492c80cca2f8d7'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:32.563238 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 782/1543] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:24:32 2022] DELETE /identity/v3/users/fc808f90939541c7a0492c80cca2f8d7 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:24:32.576109 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 762/1544] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:24:32 2022] PUT /identity/v3/projects/c3e3b33974604b8f8cdb5f4849917a4b/users/248eb53661d24b3380de06b7f6b10dd8/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:32.577077 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4398b4cc-c3ba-4db8-b50b-29800215f2b2 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:32.577506 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4398b4cc-c3ba-4db8-b50b-29800215f2b2 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:32.578032 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4398b4cc-c3ba-4db8-b50b-29800215f2b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:32.581318 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b193dc-0430-48e4-a560-f539b44a6287 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:32.581614 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b193dc-0430-48e4-a560-f539b44a6287 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:32.581930 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b193dc-0430-48e4-a560-f539b44a6287 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:32.624975 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-4398b4cc-c3ba-4db8-b50b-29800215f2b2 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:32.626265 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-4398b4cc-c3ba-4db8-b50b-29800215f2b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:32.629217 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-72b193dc-0430-48e4-a560-f539b44a6287 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:32.630398 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-72b193dc-0430-48e4-a560-f539b44a6287 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:32.680028 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:32.741432 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 763/1545] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:24:32.742482 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 783/1546] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:32.789705 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b193dc-0430-48e4-a560-f539b44a6287 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:32.791488 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-72b193dc-0430-48e4-a560-f539b44a6287 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:32.798997 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4398b4cc-c3ba-4db8-b50b-29800215f2b2 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:32.801182 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-4398b4cc-c3ba-4db8-b50b-29800215f2b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:32.882910 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b193dc-0430-48e4-a560-f539b44a6287 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:32.888122 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99c792a-6395-4eb3-b1ba-814b7f829e6f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:32.888849 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99c792a-6395-4eb3-b1ba-814b7f829e6f service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:32.889521 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99c792a-6395-4eb3-b1ba-814b7f829e6f service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:32.890072 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99c792a-6395-4eb3-b1ba-814b7f829e6f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:32.907268 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99c792a-6395-4eb3-b1ba-814b7f829e6f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:32.909546 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99c792a-6395-4eb3-b1ba-814b7f829e6f service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:32.925252 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 764/1547] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:24:32.941502 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4398b4cc-c3ba-4db8-b50b-29800215f2b2 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:32.946154 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c233e1-3937-4817-a816-e6e46e4070b4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:32.946868 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c233e1-3937-4817-a816-e6e46e4070b4 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:32.947391 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c233e1-3937-4817-a816-e6e46e4070b4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:32.947793 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c233e1-3937-4817-a816-e6e46e4070b4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:32.959983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c233e1-3937-4817-a816-e6e46e4070b4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:32.961635 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c233e1-3937-4817-a816-e6e46e4070b4 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:32.973988 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 784/1548] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 27 01:24:33.716258 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99c792a-6395-4eb3-b1ba-814b7f829e6f service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:33.718184 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e99c792a-6395-4eb3-b1ba-814b7f829e6f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:33.744750 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99c792a-6395-4eb3-b1ba-814b7f829e6f service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:33.765357 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef861ab-a728-4d92-beaf-188ff57f2398 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:33.766192 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef861ab-a728-4d92-beaf-188ff57f2398 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:33.766563 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef861ab-a728-4d92-beaf-188ff57f2398 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:33.767554 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef861ab-a728-4d92-beaf-188ff57f2398 admin admin] PATH_INFO: `/v3/projects/267a03e8dba3447baf87f211fc6fff93` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:33.769150 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fef861ab-a728-4d92-beaf-188ff57f2398 admin admin] RBAC: Authorizing `identity:delete_project(project_id=267a03e8dba3447baf87f211fc6fff93)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:33.771108 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fef861ab-a728-4d92-beaf-188ff57f2398 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:33.834705 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-fef861ab-a728-4d92-beaf-188ff57f2398 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '267a03e8dba3447baf87f211fc6fff93', 'request_id': 'req-fef861ab-a728-4d92-beaf-188ff57f2398'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:33.842297 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-fef861ab-a728-4d92-beaf-188ff57f2398 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '267a03e8dba3447baf87f211fc6fff93', 'request_id': 'req-fef861ab-a728-4d92-beaf-188ff57f2398'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:33.849339 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-fef861ab-a728-4d92-beaf-188ff57f2398 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '267a03e8dba3447baf87f211fc6fff93', 'request_id': 'req-fef861ab-a728-4d92-beaf-188ff57f2398'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:33.861774 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-fef861ab-a728-4d92-beaf-188ff57f2398 admin admin] The token cache is being invalidate because project 267a03e8dba3447baf87f211fc6fff93 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:24:33.862568 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-fef861ab-a728-4d92-beaf-188ff57f2398 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:33.863244 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-fef861ab-a728-4d92-beaf-188ff57f2398 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:33.865040 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 765/1549] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:24:33 2022] DELETE /identity/v3/projects/267a03e8dba3447baf87f211fc6fff93 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:33.869392 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c233e1-3937-4817-a816-e6e46e4070b4 service neutron] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:33.881159 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c233e1-3937-4817-a816-e6e46e4070b4 service neutron] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:34.002070 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-979644aa-2c92-40a1-b428-3148688d50d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:34.011986 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979644aa-2c92-40a1-b428-3148688d50d0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:34.020200 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979644aa-2c92-40a1-b428-3148688d50d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:34.021383 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979644aa-2c92-40a1-b428-3148688d50d0 admin admin] PATH_INFO: `/v3/users/b106f043ba8649b9a36941d09a3e1ec6` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:34.027009 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-979644aa-2c92-40a1-b428-3148688d50d0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b106f043ba8649b9a36941d09a3e1ec6)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:34.030382 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-979644aa-2c92-40a1-b428-3148688d50d0 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:34.100531 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef861ab-a728-4d92-beaf-188ff57f2398 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:34.102273 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-fef861ab-a728-4d92-beaf-188ff57f2398 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:34.121888 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:34.132235 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef861ab-a728-4d92-beaf-188ff57f2398 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:34.142616 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1804554f-2e2b-41f9-a153-53c35ad13698 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:34.143291 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1804554f-2e2b-41f9-a153-53c35ad13698 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:34.143604 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1804554f-2e2b-41f9-a153-53c35ad13698 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:34.143979 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1804554f-2e2b-41f9-a153-53c35ad13698 admin admin] PATH_INFO: `/v3/projects/bf79533a12114d758249ef92f8a11a71` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:34.145359 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1804554f-2e2b-41f9-a153-53c35ad13698 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bf79533a12114d758249ef92f8a11a71)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:34.147791 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-979644aa-2c92-40a1-b428-3148688d50d0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b106f043ba8649b9a36941d09a3e1ec6'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:34.147938 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1804554f-2e2b-41f9-a153-53c35ad13698 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:34.162532 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-979644aa-2c92-40a1-b428-3148688d50d0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b106f043ba8649b9a36941d09a3e1ec6'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:34.188641 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-979644aa-2c92-40a1-b428-3148688d50d0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b106f043ba8649b9a36941d09a3e1ec6'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:34.189602 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-979644aa-2c92-40a1-b428-3148688d50d0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b106f043ba8649b9a36941d09a3e1ec6'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:34.190883 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-979644aa-2c92-40a1-b428-3148688d50d0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b106f043ba8649b9a36941d09a3e1ec6'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:34.192798 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:34.213732 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 785/1550] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:24:33 2022] DELETE /identity/v3/users/b106f043ba8649b9a36941d09a3e1ec6 => generated 0 bytes in 345 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:34.393000 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1804554f-2e2b-41f9-a153-53c35ad13698 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bf79533a12114d758249ef92f8a11a71', 'request_id': 'req-1804554f-2e2b-41f9-a153-53c35ad13698'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:34.411161 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1804554f-2e2b-41f9-a153-53c35ad13698 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bf79533a12114d758249ef92f8a11a71', 'request_id': 'req-1804554f-2e2b-41f9-a153-53c35ad13698'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:34.429486 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1804554f-2e2b-41f9-a153-53c35ad13698 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bf79533a12114d758249ef92f8a11a71', 'request_id': 'req-1804554f-2e2b-41f9-a153-53c35ad13698'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:34.450645 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1804554f-2e2b-41f9-a153-53c35ad13698 admin admin] The token cache is being invalidate because project bf79533a12114d758249ef92f8a11a71 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:24:34.450946 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1804554f-2e2b-41f9-a153-53c35ad13698 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:34.455137 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1804554f-2e2b-41f9-a153-53c35ad13698 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:34.463855 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-979644aa-2c92-40a1-b428-3148688d50d0 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:34.465656 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-979644aa-2c92-40a1-b428-3148688d50d0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:34.465776 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 766/1551] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:24:34 2022] DELETE /identity/v3/projects/bf79533a12114d758249ef92f8a11a71 => generated 0 bytes in 360 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:24:34.571071 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-979644aa-2c92-40a1-b428-3148688d50d0 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:34.704039 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2674d17a-22c7-4565-9d63-66f6e6903a9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:34.706787 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2674d17a-22c7-4565-9d63-66f6e6903a9f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:34.707187 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2674d17a-22c7-4565-9d63-66f6e6903a9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:34.707585 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2674d17a-22c7-4565-9d63-66f6e6903a9f admin admin] PATH_INFO: `/v3/users/2ca19a201243468583cf1b44beacd989` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:34.709906 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2674d17a-22c7-4565-9d63-66f6e6903a9f admin admin] RBAC: Authorizing `identity:delete_user(user_id=2ca19a201243468583cf1b44beacd989)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:34.715909 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2674d17a-22c7-4565-9d63-66f6e6903a9f admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:34.786170 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-2674d17a-22c7-4565-9d63-66f6e6903a9f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2ca19a201243468583cf1b44beacd989'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:34.807786 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-2674d17a-22c7-4565-9d63-66f6e6903a9f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2ca19a201243468583cf1b44beacd989'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:34.828321 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-2674d17a-22c7-4565-9d63-66f6e6903a9f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2ca19a201243468583cf1b44beacd989'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:34.830050 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-2674d17a-22c7-4565-9d63-66f6e6903a9f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2ca19a201243468583cf1b44beacd989'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:34.830905 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-2674d17a-22c7-4565-9d63-66f6e6903a9f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2ca19a201243468583cf1b44beacd989'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:34.853104 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 786/1552] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:24:34 2022] DELETE /identity/v3/users/2ca19a201243468583cf1b44beacd989 => generated 0 bytes in 389 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:24:34.905606 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6002bd02-ba9d-4563-a9bb-101fc50fe921 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:34.906258 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6002bd02-ba9d-4563-a9bb-101fc50fe921 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:34.906601 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6002bd02-ba9d-4563-a9bb-101fc50fe921 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:34.948964 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2674d17a-22c7-4565-9d63-66f6e6903a9f admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:34.950482 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-2674d17a-22c7-4565-9d63-66f6e6903a9f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:34.984329 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-6002bd02-ba9d-4563-a9bb-101fc50fe921 None None] MFA Rules not processed for user `248eb53661d24b3380de06b7f6b10dd8`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:34.985656 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-6002bd02-ba9d-4563-a9bb-101fc50fe921 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:34.998346 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:35.097886 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2674d17a-22c7-4565-9d63-66f6e6903a9f admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:35.113126 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 767/1553] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:34 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:35.113714 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e4becb-038b-4e19-a7a4-46fdd473bb62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:35.113714 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e4becb-038b-4e19-a7a4-46fdd473bb62 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:35.113714 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e4becb-038b-4e19-a7a4-46fdd473bb62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:35.113714 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e4becb-038b-4e19-a7a4-46fdd473bb62 admin admin] PATH_INFO: `/v3/projects/e991555393b147c594f35fabd0c8ba4f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:35.114910 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6002bd02-ba9d-4563-a9bb-101fc50fe921 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:35.116359 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e4becb-038b-4e19-a7a4-46fdd473bb62 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e991555393b147c594f35fabd0c8ba4f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:35.116482 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-6002bd02-ba9d-4563-a9bb-101fc50fe921 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:35.118496 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e4becb-038b-4e19-a7a4-46fdd473bb62 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:35.130693 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6002bd02-ba9d-4563-a9bb-101fc50fe921 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:35.136049 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9402a3f-4c63-4cca-84de-e63f3ed14f8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:35.137420 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9402a3f-4c63-4cca-84de-e63f3ed14f8c admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:35.138265 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9402a3f-4c63-4cca-84de-e63f3ed14f8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:35.138728 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9402a3f-4c63-4cca-84de-e63f3ed14f8c admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:35.139562 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9402a3f-4c63-4cca-84de-e63f3ed14f8c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:35.143143 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9402a3f-4c63-4cca-84de-e63f3ed14f8c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:35.198150 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 768/1554] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:24:35 2022] POST /identity/v3/projects => generated 373 bytes in 84 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:24:35.200469 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9402a3f-4c63-4cca-84de-e63f3ed14f8c admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:35.222433 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-f8e4becb-038b-4e19-a7a4-46fdd473bb62 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e991555393b147c594f35fabd0c8ba4f', 'request_id': 'req-f8e4becb-038b-4e19-a7a4-46fdd473bb62'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:35.244941 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-f8e4becb-038b-4e19-a7a4-46fdd473bb62 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e991555393b147c594f35fabd0c8ba4f', 'request_id': 'req-f8e4becb-038b-4e19-a7a4-46fdd473bb62'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:35.256795 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-f8e4becb-038b-4e19-a7a4-46fdd473bb62 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e991555393b147c594f35fabd0c8ba4f', 'request_id': 'req-f8e4becb-038b-4e19-a7a4-46fdd473bb62'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:35.276699 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-f8e4becb-038b-4e19-a7a4-46fdd473bb62 admin admin] The token cache is being invalidate because project e991555393b147c594f35fabd0c8ba4f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:24:35.277118 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-f8e4becb-038b-4e19-a7a4-46fdd473bb62 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:35.277839 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-f8e4becb-038b-4e19-a7a4-46fdd473bb62 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:35.279568 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 787/1555] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:24:34 2022] DELETE /identity/v3/projects/e991555393b147c594f35fabd0c8ba4f => generated 0 bytes in 331 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:35.285349 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4265abf-dfbb-4207-905f-f62121bbb33b None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:35.285748 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4265abf-dfbb-4207-905f-f62121bbb33b None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:35.286140 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4265abf-dfbb-4207-905f-f62121bbb33b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:35.308741 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:35.310152 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-d4265abf-dfbb-4207-905f-f62121bbb33b None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:35.311510 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-d4265abf-dfbb-4207-905f-f62121bbb33b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:35.317778 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9402a3f-4c63-4cca-84de-e63f3ed14f8c admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:35.413264 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f779a4-0aa8-4d78-8d74-4be8d356663a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:35.413990 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f779a4-0aa8-4d78-8d74-4be8d356663a admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:35.414186 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 788/1556] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:35.414325 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f779a4-0aa8-4d78-8d74-4be8d356663a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:35.414602 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f779a4-0aa8-4d78-8d74-4be8d356663a admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:35.414957 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4265abf-dfbb-4207-905f-f62121bbb33b None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:35.415247 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f779a4-0aa8-4d78-8d74-4be8d356663a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:35.416363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-d4265abf-dfbb-4207-905f-f62121bbb33b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:35.417310 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f779a4-0aa8-4d78-8d74-4be8d356663a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:35.431303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4265abf-dfbb-4207-905f-f62121bbb33b None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:35.436943 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eda4f86-0a62-4993-8a1a-66573b4e4e9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:35.437673 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eda4f86-0a62-4993-8a1a-66573b4e4e9c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:35.437994 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eda4f86-0a62-4993-8a1a-66573b4e4e9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:35.438274 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eda4f86-0a62-4993-8a1a-66573b4e4e9c admin admin] PATH_INFO: `/v3/projects/0eb5ac85ccde4e08bdd70cd477fc0bb3` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:35.439771 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eda4f86-0a62-4993-8a1a-66573b4e4e9c admin admin] RBAC: Authorizing `identity:delete_project(project_id=0eb5ac85ccde4e08bdd70cd477fc0bb3)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:35.442859 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eda4f86-0a62-4993-8a1a-66573b4e4e9c admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:35.446446 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 769/1557] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:24:35 2022] POST /identity/v3/users => generated 293 bytes in 247 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:24:35.448138 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef63d79a-005f-49f2-b9a1-11d4fa98bcd1 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:35.448628 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef63d79a-005f-49f2-b9a1-11d4fa98bcd1 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:35.449131 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef63d79a-005f-49f2-b9a1-11d4fa98bcd1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:35.471908 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-ef63d79a-005f-49f2-b9a1-11d4fa98bcd1 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:35.473293 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ef63d79a-005f-49f2-b9a1-11d4fa98bcd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:35.482602 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 770/1558] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:35.483418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef63d79a-005f-49f2-b9a1-11d4fa98bcd1 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:35.484773 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ef63d79a-005f-49f2-b9a1-11d4fa98bcd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:35.507849 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef63d79a-005f-49f2-b9a1-11d4fa98bcd1 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:35.513340 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c1be7d-c78e-4568-a016-d5884b4a0fef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:35.514188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c1be7d-c78e-4568-a016-d5884b4a0fef admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:35.514572 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c1be7d-c78e-4568-a016-d5884b4a0fef admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:35.514846 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c1be7d-c78e-4568-a016-d5884b4a0fef admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:35.516128 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c1be7d-c78e-4568-a016-d5884b4a0fef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:35.517844 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c1be7d-c78e-4568-a016-d5884b4a0fef admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:35.542082 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 771/1559] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:35 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:35.543390 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c1be7d-c78e-4568-a016-d5884b4a0fef admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:35.557323 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c1be7d-c78e-4568-a016-d5884b4a0fef admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:35.563163 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eff76b0-1a6e-4be0-bf62-3e8b7dcce2fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:35.563777 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eff76b0-1a6e-4be0-bf62-3e8b7dcce2fd admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:35.564051 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eff76b0-1a6e-4be0-bf62-3e8b7dcce2fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:35.564319 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eff76b0-1a6e-4be0-bf62-3e8b7dcce2fd admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:35.565002 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eff76b0-1a6e-4be0-bf62-3e8b7dcce2fd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:24:35.565436 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eff76b0-1a6e-4be0-bf62-3e8b7dcce2fd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:35.567309 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eff76b0-1a6e-4be0-bf62-3e8b7dcce2fd admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:35.581228 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 772/1560] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:24:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:24:35.582163 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eff76b0-1a6e-4be0-bf62-3e8b7dcce2fd admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:35.594753 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eff76b0-1a6e-4be0-bf62-3e8b7dcce2fd admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:35.600526 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f2ba03-49b9-4583-a3db-005e7be78df6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:35.601233 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f2ba03-49b9-4583-a3db-005e7be78df6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:35.601512 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f2ba03-49b9-4583-a3db-005e7be78df6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:35.601785 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f2ba03-49b9-4583-a3db-005e7be78df6 admin admin] PATH_INFO: `/v3/projects/f6f36b70f1774b18b5472730833fd38c/users/6bd65bfc7c994b58a47b2314bedd8c84/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:35.618776 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f2ba03-49b9-4583-a3db-005e7be78df6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f6f36b70f1774b18b5472730833fd38c, user_id=6bd65bfc7c994b58a47b2314bedd8c84, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:35.621483 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f2ba03-49b9-4583-a3db-005e7be78df6 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:35.635689 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 773/1561] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:24:35 2022] PUT /identity/v3/projects/f6f36b70f1774b18b5472730833fd38c/users/6bd65bfc7c994b58a47b2314bedd8c84/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:35.636696 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f2ba03-49b9-4583-a3db-005e7be78df6 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:35.642365 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:35.650347 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-0eda4f86-0a62-4993-8a1a-66573b4e4e9c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0eb5ac85ccde4e08bdd70cd477fc0bb3', 'request_id': 'req-0eda4f86-0a62-4993-8a1a-66573b4e4e9c'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:35.658504 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-0eda4f86-0a62-4993-8a1a-66573b4e4e9c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0eb5ac85ccde4e08bdd70cd477fc0bb3', 'request_id': 'req-0eda4f86-0a62-4993-8a1a-66573b4e4e9c'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:35.672402 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-0eda4f86-0a62-4993-8a1a-66573b4e4e9c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0eb5ac85ccde4e08bdd70cd477fc0bb3', 'request_id': 'req-0eda4f86-0a62-4993-8a1a-66573b4e4e9c'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:35.684348 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-0eda4f86-0a62-4993-8a1a-66573b4e4e9c admin admin] The token cache is being invalidate because project 0eb5ac85ccde4e08bdd70cd477fc0bb3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:24:35.684635 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-0eda4f86-0a62-4993-8a1a-66573b4e4e9c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:35.685351 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-0eda4f86-0a62-4993-8a1a-66573b4e4e9c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:35.686889 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 789/1562] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:24:35 2022] DELETE /identity/v3/projects/0eb5ac85ccde4e08bdd70cd477fc0bb3 => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:35.687794 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eda4f86-0a62-4993-8a1a-66573b4e4e9c admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:35.689132 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-0eda4f86-0a62-4993-8a1a-66573b4e4e9c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:35.716717 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f2ba03-49b9-4583-a3db-005e7be78df6 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:35.794472 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:35.824515 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eda4f86-0a62-4993-8a1a-66573b4e4e9c admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:35.834323 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64675357-c18d-47bc-b32c-cae259e5f0a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:35.835215 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64675357-c18d-47bc-b32c-cae259e5f0a9 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:35.836068 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64675357-c18d-47bc-b32c-cae259e5f0a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:35.836312 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64675357-c18d-47bc-b32c-cae259e5f0a9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:35.837027 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64675357-c18d-47bc-b32c-cae259e5f0a9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:35.839664 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64675357-c18d-47bc-b32c-cae259e5f0a9 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:35.842064 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ca97d6-c26e-454a-9d9b-3075e3e85cb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:35.843503 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ca97d6-c26e-454a-9d9b-3075e3e85cb9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:35.843792 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ca97d6-c26e-454a-9d9b-3075e3e85cb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:35.845414 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ca97d6-c26e-454a-9d9b-3075e3e85cb9 admin admin] PATH_INFO: `/v3/users/b6def6d4027a45eb8b40c3db72fda121` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:35.854216 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38ca97d6-c26e-454a-9d9b-3075e3e85cb9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b6def6d4027a45eb8b40c3db72fda121)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:35.856311 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38ca97d6-c26e-454a-9d9b-3075e3e85cb9 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:35.878537 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 774/1563] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:24:35 2022] POST /identity/v3/projects => generated 377 bytes in 242 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:24:35.879582 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64675357-c18d-47bc-b32c-cae259e5f0a9 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:35.881871 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-64675357-c18d-47bc-b32c-cae259e5f0a9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:35.921127 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-38ca97d6-c26e-454a-9d9b-3075e3e85cb9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b6def6d4027a45eb8b40c3db72fda121'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:35.937781 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-38ca97d6-c26e-454a-9d9b-3075e3e85cb9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b6def6d4027a45eb8b40c3db72fda121'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:35.963317 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-38ca97d6-c26e-454a-9d9b-3075e3e85cb9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b6def6d4027a45eb8b40c3db72fda121'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:35.964070 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-38ca97d6-c26e-454a-9d9b-3075e3e85cb9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b6def6d4027a45eb8b40c3db72fda121'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:35.964814 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-38ca97d6-c26e-454a-9d9b-3075e3e85cb9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b6def6d4027a45eb8b40c3db72fda121'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:35.984662 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 790/1564] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:24:35 2022] DELETE /identity/v3/users/b6def6d4027a45eb8b40c3db72fda121 => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:35.987983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b2abc9-2f87-4096-896a-5acdfe842806 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:35.987983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b2abc9-2f87-4096-896a-5acdfe842806 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:35.988487 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b2abc9-2f87-4096-896a-5acdfe842806 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:36.001758 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64675357-c18d-47bc-b32c-cae259e5f0a9 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:36.010734 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-a1b2abc9-2f87-4096-896a-5acdfe842806 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:36.012272 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-a1b2abc9-2f87-4096-896a-5acdfe842806 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:36.091902 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801615a1-6d4f-49f1-b5b4-5d2fa24a975a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:36.093039 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801615a1-6d4f-49f1-b5b4-5d2fa24a975a admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:36.093582 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801615a1-6d4f-49f1-b5b4-5d2fa24a975a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:36.094259 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801615a1-6d4f-49f1-b5b4-5d2fa24a975a admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:36.095099 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-801615a1-6d4f-49f1-b5b4-5d2fa24a975a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:36.099641 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-801615a1-6d4f-49f1-b5b4-5d2fa24a975a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:36.104695 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:36.119044 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 775/1565] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:35 2022] GET /identity/v3/roles => generated 1420 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:36.120893 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801615a1-6d4f-49f1-b5b4-5d2fa24a975a admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:36.123450 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-801615a1-6d4f-49f1-b5b4-5d2fa24a975a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:36.137732 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 791/1566] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:36.139035 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0df4a50-59c5-4e96-824a-5e48e408adb7 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:36.139333 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0df4a50-59c5-4e96-824a-5e48e408adb7 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:36.139591 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0df4a50-59c5-4e96-824a-5e48e408adb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:36.143441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801615a1-6d4f-49f1-b5b4-5d2fa24a975a admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:36.149354 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53904a6-37b9-4a93-94ef-181ea20fab7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:36.151100 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53904a6-37b9-4a93-94ef-181ea20fab7b admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:36.152029 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53904a6-37b9-4a93-94ef-181ea20fab7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:36.152482 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53904a6-37b9-4a93-94ef-181ea20fab7b admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:36.153347 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b53904a6-37b9-4a93-94ef-181ea20fab7b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:36.156137 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b53904a6-37b9-4a93-94ef-181ea20fab7b admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:36.162750 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-a0df4a50-59c5-4e96-824a-5e48e408adb7 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:36.164062 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-a0df4a50-59c5-4e96-824a-5e48e408adb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:36.175128 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 792/1567] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:36.176043 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0df4a50-59c5-4e96-824a-5e48e408adb7 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:36.177739 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-a0df4a50-59c5-4e96-824a-5e48e408adb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:36.197541 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0df4a50-59c5-4e96-824a-5e48e408adb7 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:36.197740 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 776/1568] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:24:36 2022] POST /identity/v3/users => generated 289 bytes in 77 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:24:36.199865 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b2339f-3052-4edd-bf78-e66cd0d827b9 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:36.200151 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b2339f-3052-4edd-bf78-e66cd0d827b9 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:36.201325 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b2339f-3052-4edd-bf78-e66cd0d827b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:36.209255 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7cf85ef-a354-4ac8-be9b-46bcdee2c3d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:36.209935 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7cf85ef-a354-4ac8-be9b-46bcdee2c3d3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:36.210255 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7cf85ef-a354-4ac8-be9b-46bcdee2c3d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:36.210471 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7cf85ef-a354-4ac8-be9b-46bcdee2c3d3 admin admin] PATH_INFO: `/v3/projects/f6f36b70f1774b18b5472730833fd38c/users/6bd65bfc7c994b58a47b2314bedd8c84/roles/414286927fdd44688107ff5e10235acf` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:36.213256 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7cf85ef-a354-4ac8-be9b-46bcdee2c3d3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f6f36b70f1774b18b5472730833fd38c, user_id=6bd65bfc7c994b58a47b2314bedd8c84, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:36.216624 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7cf85ef-a354-4ac8-be9b-46bcdee2c3d3 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:36.233285 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 793/1569] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:24:36 2022] PUT /identity/v3/projects/f6f36b70f1774b18b5472730833fd38c/users/6bd65bfc7c994b58a47b2314bedd8c84/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:36.234302 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7cf85ef-a354-4ac8-be9b-46bcdee2c3d3 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:36.237881 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-c8b2339f-3052-4edd-bf78-e66cd0d827b9 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:36.240670 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-c8b2339f-3052-4edd-bf78-e66cd0d827b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:36.313529 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:36.336941 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 777/1570] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:36.338101 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b2339f-3052-4edd-bf78-e66cd0d827b9 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:36.341593 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-c8b2339f-3052-4edd-bf78-e66cd0d827b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:36.349041 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7cf85ef-a354-4ac8-be9b-46bcdee2c3d3 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:36.364995 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c177999-037d-4b13-95c8-3c33e3f1ae0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:36.365847 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c177999-037d-4b13-95c8-3c33e3f1ae0e admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:36.366127 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c177999-037d-4b13-95c8-3c33e3f1ae0e admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:36.366403 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c177999-037d-4b13-95c8-3c33e3f1ae0e admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:36.369315 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c177999-037d-4b13-95c8-3c33e3f1ae0e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:36.371289 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c177999-037d-4b13-95c8-3c33e3f1ae0e admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:36.402356 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 794/1571] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:36 2022] GET /identity/v3/roles => generated 1420 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:36.403072 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c177999-037d-4b13-95c8-3c33e3f1ae0e admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:36.424250 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c177999-037d-4b13-95c8-3c33e3f1ae0e admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:36.440630 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b945b5-e921-40b8-afc6-3c0dda5fa986 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:36.441778 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b945b5-e921-40b8-afc6-3c0dda5fa986 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:36.442259 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b945b5-e921-40b8-afc6-3c0dda5fa986 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:36.442706 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b945b5-e921-40b8-afc6-3c0dda5fa986 admin admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:36.443380 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b2339f-3052-4edd-bf78-e66cd0d827b9 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:36.443740 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91b945b5-e921-40b8-afc6-3c0dda5fa986 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:24:36.444238 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91b945b5-e921-40b8-afc6-3c0dda5fa986 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:36.447157 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91b945b5-e921-40b8-afc6-3c0dda5fa986 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:36.447345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5464703d-27e6-4897-a492-9af20c8231d6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:36.447919 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5464703d-27e6-4897-a492-9af20c8231d6 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:36.448196 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5464703d-27e6-4897-a492-9af20c8231d6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:36.448463 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5464703d-27e6-4897-a492-9af20c8231d6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:36.462931 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5464703d-27e6-4897-a492-9af20c8231d6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:36.465364 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5464703d-27e6-4897-a492-9af20c8231d6 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:36.470707 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 795/1572] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:24:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:24:36.470707 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b945b5-e921-40b8-afc6-3c0dda5fa986 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:36.483035 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 778/1573] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:24:36.484453 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5464703d-27e6-4897-a492-9af20c8231d6 service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:36.485031 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b945b5-e921-40b8-afc6-3c0dda5fa986 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:36.492795 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef1d8bc-6ba0-4ae4-9df1-4ea96559c9df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:36.493581 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef1d8bc-6ba0-4ae4-9df1-4ea96559c9df admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:36.493943 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef1d8bc-6ba0-4ae4-9df1-4ea96559c9df admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:36.494286 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef1d8bc-6ba0-4ae4-9df1-4ea96559c9df admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:36.494994 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef1d8bc-6ba0-4ae4-9df1-4ea96559c9df admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:36.497994 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef1d8bc-6ba0-4ae4-9df1-4ea96559c9df admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:36.498114 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5464703d-27e6-4897-a492-9af20c8231d6 service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:36.505546 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6c993b-b1b4-464b-9327-f53482997a0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:36.507153 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6c993b-b1b4-464b-9327-f53482997a0e admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:36.507695 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6c993b-b1b4-464b-9327-f53482997a0e admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:36.508137 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6c993b-b1b4-464b-9327-f53482997a0e admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:36.508999 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6c993b-b1b4-464b-9327-f53482997a0e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:36.511419 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6c993b-b1b4-464b-9327-f53482997a0e admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:36.517829 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 796/1574] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:36 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:36.519696 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef1d8bc-6ba0-4ae4-9df1-4ea96559c9df admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:36.535543 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef1d8bc-6ba0-4ae4-9df1-4ea96559c9df admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:36.545502 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d6f028-9aaf-4bb4-b665-3b62244382b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:36.547340 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d6f028-9aaf-4bb4-b665-3b62244382b4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:36.547340 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d6f028-9aaf-4bb4-b665-3b62244382b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:36.547340 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d6f028-9aaf-4bb4-b665-3b62244382b4 admin admin] PATH_INFO: `/v3/projects/872936a3af034418b87ab13cfbe191c1/users/0960e7d937cb4157a81d288e98d81f8f/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:36.565828 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 779/1575] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:24:36 2022] POST /identity/v3/projects => generated 371 bytes in 82 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:24:36.571008 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6c993b-b1b4-464b-9327-f53482997a0e admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:36.572134 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76d6f028-9aaf-4bb4-b665-3b62244382b4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=872936a3af034418b87ab13cfbe191c1, user_id=0960e7d937cb4157a81d288e98d81f8f, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:36.574892 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76d6f028-9aaf-4bb4-b665-3b62244382b4 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:36.613145 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 797/1576] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:24:36 2022] PUT /identity/v3/projects/872936a3af034418b87ab13cfbe191c1/users/0960e7d937cb4157a81d288e98d81f8f/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:36.616986 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cff88fb-7732-480c-8fd0-7fcd40c991b4 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:36.617468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cff88fb-7732-480c-8fd0-7fcd40c991b4 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:36.617836 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cff88fb-7732-480c-8fd0-7fcd40c991b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:36.651621 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-3cff88fb-7732-480c-8fd0-7fcd40c991b4 None None] MFA Rules not processed for user `0960e7d937cb4157a81d288e98d81f8f`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:36.652963 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-3cff88fb-7732-480c-8fd0-7fcd40c991b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:36.699801 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6c993b-b1b4-464b-9327-f53482997a0e admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:36.819663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718b8324-fcc7-44c8-ac88-93aeae2fdbf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:36.820761 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718b8324-fcc7-44c8-ac88-93aeae2fdbf5 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:36.821260 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718b8324-fcc7-44c8-ac88-93aeae2fdbf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:36.821746 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718b8324-fcc7-44c8-ac88-93aeae2fdbf5 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:36.822715 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718b8324-fcc7-44c8-ac88-93aeae2fdbf5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:36.824907 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718b8324-fcc7-44c8-ac88-93aeae2fdbf5 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:36.832387 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 798/1577] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:36 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:24:36.834392 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cff88fb-7732-480c-8fd0-7fcd40c991b4 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:36.849147 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cff88fb-7732-480c-8fd0-7fcd40c991b4 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:36.852764 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 780/1578] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:24:36 2022] POST /identity/v3/users => generated 286 bytes in 282 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:24:36.857995 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718b8324-fcc7-44c8-ac88-93aeae2fdbf5 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:36.860337 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0678e111-0180-47e5-8233-c256d3894202 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:36.861336 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0678e111-0180-47e5-8233-c256d3894202 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:36.862548 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0678e111-0180-47e5-8233-c256d3894202 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:36.863010 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0678e111-0180-47e5-8233-c256d3894202 admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:36.864138 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0678e111-0180-47e5-8233-c256d3894202 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:36.867508 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0678e111-0180-47e5-8233-c256d3894202 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:36.870495 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718b8324-fcc7-44c8-ac88-93aeae2fdbf5 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:36.874277 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:36.876481 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56181d6-3baa-4a71-b5cf-e52e7043838c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:36.877149 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56181d6-3baa-4a71-b5cf-e52e7043838c admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:36.877436 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56181d6-3baa-4a71-b5cf-e52e7043838c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:36.878346 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56181d6-3baa-4a71-b5cf-e52e7043838c admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:36.879584 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56181d6-3baa-4a71-b5cf-e52e7043838c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:36.880993 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56181d6-3baa-4a71-b5cf-e52e7043838c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:36.902158 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 781/1579] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:36 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:36.906996 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56181d6-3baa-4a71-b5cf-e52e7043838c admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:36.913222 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:36.914923 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 799/1580] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:24:36 2022] POST /identity/v3/projects => generated 377 bytes in 81 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:24:36.918912 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0678e111-0180-47e5-8233-c256d3894202 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:36.922642 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56181d6-3baa-4a71-b5cf-e52e7043838c admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:37.012824 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0678e111-0180-47e5-8233-c256d3894202 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:37.019557 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210e958c-fa3d-42d6-bfb3-83c6bbf19b33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:37.020593 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210e958c-fa3d-42d6-bfb3-83c6bbf19b33 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:37.021560 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210e958c-fa3d-42d6-bfb3-83c6bbf19b33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:37.021560 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210e958c-fa3d-42d6-bfb3-83c6bbf19b33 admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:37.021833 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-210e958c-fa3d-42d6-bfb3-83c6bbf19b33 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:37.024042 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-210e958c-fa3d-42d6-bfb3-83c6bbf19b33 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:37.057304 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 800/1581] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:24:36 2022] POST /identity/v3/users => generated 295 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:24:37.057706 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67a6969-95bd-4e78-8152-15ef66deb2e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:37.058638 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67a6969-95bd-4e78-8152-15ef66deb2e8 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:37.059120 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67a6969-95bd-4e78-8152-15ef66deb2e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:37.059583 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67a6969-95bd-4e78-8152-15ef66deb2e8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:37.060915 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c67a6969-95bd-4e78-8152-15ef66deb2e8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:37.061146 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210e958c-fa3d-42d6-bfb3-83c6bbf19b33 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:37.063523 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c67a6969-95bd-4e78-8152-15ef66deb2e8 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:37.072137 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210e958c-fa3d-42d6-bfb3-83c6bbf19b33 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:37.079898 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0930c368-c8a8-4bff-9a8c-040489b4bd1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:37.080988 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0930c368-c8a8-4bff-9a8c-040489b4bd1f admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:37.080988 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0930c368-c8a8-4bff-9a8c-040489b4bd1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:37.081347 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0930c368-c8a8-4bff-9a8c-040489b4bd1f admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:37.082080 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0930c368-c8a8-4bff-9a8c-040489b4bd1f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:37.082524 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 782/1582] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:36 2022] GET /identity/v3/roles => generated 1420 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:37.084218 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0930c368-c8a8-4bff-9a8c-040489b4bd1f admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:37.086257 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67a6969-95bd-4e78-8152-15ef66deb2e8 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:37.097230 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 801/1583] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:37 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:37.100848 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67a6969-95bd-4e78-8152-15ef66deb2e8 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:37.100991 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0930c368-c8a8-4bff-9a8c-040489b4bd1f admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:37.106034 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84d8297-ee09-4498-8150-3f8e9c144f8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:37.106712 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84d8297-ee09-4498-8150-3f8e9c144f8e admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:37.106979 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84d8297-ee09-4498-8150-3f8e9c144f8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:37.107305 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84d8297-ee09-4498-8150-3f8e9c144f8e admin admin] PATH_INFO: `/v3/projects/86efb04a71c7402d87d34ca096d8d0fa/users/2b2f77ca967a48e1aec54a4771ec0df0/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:37.114581 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0930c368-c8a8-4bff-9a8c-040489b4bd1f admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:37.122170 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2aac8e-64fc-43dd-beea-801fc8f457f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:37.123297 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2aac8e-64fc-43dd-beea-801fc8f457f9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:37.123599 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2aac8e-64fc-43dd-beea-801fc8f457f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:37.123954 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2aac8e-64fc-43dd-beea-801fc8f457f9 admin admin] PATH_INFO: `/v3/projects/652cb71a72974636bd7ef43ad556fcb6/users/189b9109149d4a01a9a3dda68908e03d/roles/414286927fdd44688107ff5e10235acf` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:37.129606 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e84d8297-ee09-4498-8150-3f8e9c144f8e admin admin] RBAC: Authorizing `identity:create_grant(project_id=86efb04a71c7402d87d34ca096d8d0fa, user_id=2b2f77ca967a48e1aec54a4771ec0df0, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:37.134301 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e84d8297-ee09-4498-8150-3f8e9c144f8e admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:37.147248 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d2aac8e-64fc-43dd-beea-801fc8f457f9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=652cb71a72974636bd7ef43ad556fcb6, user_id=189b9109149d4a01a9a3dda68908e03d, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:37.150735 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d2aac8e-64fc-43dd-beea-801fc8f457f9 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:37.156952 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 783/1584] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:24:37 2022] PUT /identity/v3/projects/86efb04a71c7402d87d34ca096d8d0fa/users/2b2f77ca967a48e1aec54a4771ec0df0/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:24:37.169391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10148ef1-5031-4992-9710-465561d0772c None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:37.169795 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10148ef1-5031-4992-9710-465561d0772c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:37.170288 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10148ef1-5031-4992-9710-465561d0772c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:37.179208 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 802/1585] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:24:37 2022] PUT /identity/v3/projects/652cb71a72974636bd7ef43ad556fcb6/users/189b9109149d4a01a9a3dda68908e03d/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:24:37.183233 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2aac8e-64fc-43dd-beea-801fc8f457f9 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:37.213609 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-10148ef1-5031-4992-9710-465561d0772c None None] MFA Rules not processed for user `2b2f77ca967a48e1aec54a4771ec0df0`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:37.215411 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-10148ef1-5031-4992-9710-465561d0772c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:37.285541 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2aac8e-64fc-43dd-beea-801fc8f457f9 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:37.291833 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478e8f3a-0c3c-4370-9256-701221ccc680 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:37.292672 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478e8f3a-0c3c-4370-9256-701221ccc680 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:37.293188 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478e8f3a-0c3c-4370-9256-701221ccc680 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:37.293660 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478e8f3a-0c3c-4370-9256-701221ccc680 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:37.294478 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478e8f3a-0c3c-4370-9256-701221ccc680 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:37.296394 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478e8f3a-0c3c-4370-9256-701221ccc680 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:37.306466 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 784/1586] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:37 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:37.307583 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10148ef1-5031-4992-9710-465561d0772c None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:37.314247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 803/1587] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:37 2022] GET /identity/v3/roles => generated 1420 bytes in 131 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 27 01:24:37.316032 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478e8f3a-0c3c-4370-9256-701221ccc680 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:37.323733 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10148ef1-5031-4992-9710-465561d0772c None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:37.330646 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea0138d-f123-4b5f-8081-ad20bac6f2f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:37.330969 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478e8f3a-0c3c-4370-9256-701221ccc680 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:37.331652 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea0138d-f123-4b5f-8081-ad20bac6f2f3 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:37.332035 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea0138d-f123-4b5f-8081-ad20bac6f2f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:37.332413 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea0138d-f123-4b5f-8081-ad20bac6f2f3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:37.333363 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea0138d-f123-4b5f-8081-ad20bac6f2f3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:37.335785 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea0138d-f123-4b5f-8081-ad20bac6f2f3 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:37.337498 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e364948-7c8f-4a0a-848a-967cab987b67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:37.338504 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e364948-7c8f-4a0a-848a-967cab987b67 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:37.338936 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e364948-7c8f-4a0a-848a-967cab987b67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:37.341093 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e364948-7c8f-4a0a-848a-967cab987b67 admin admin] PATH_INFO: `/v3/projects/652cb71a72974636bd7ef43ad556fcb6/users/189b9109149d4a01a9a3dda68908e03d/roles/414286927fdd44688107ff5e10235acf` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:37.343497 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e364948-7c8f-4a0a-848a-967cab987b67 admin admin] RBAC: Authorizing `identity:create_grant(project_id=652cb71a72974636bd7ef43ad556fcb6, user_id=189b9109149d4a01a9a3dda68908e03d, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:37.348158 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e364948-7c8f-4a0a-848a-967cab987b67 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:37.360731 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 804/1588] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:24:37 2022] PUT /identity/v3/projects/652cb71a72974636bd7ef43ad556fcb6/users/189b9109149d4a01a9a3dda68908e03d/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:37.375237 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e985657-4554-4d15-b704-0e8e1472b0c8 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:37.375903 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e985657-4554-4d15-b704-0e8e1472b0c8 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:37.376411 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e985657-4554-4d15-b704-0e8e1472b0c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:37.417295 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-6e985657-4554-4d15-b704-0e8e1472b0c8 None None] MFA Rules not processed for user `189b9109149d4a01a9a3dda68908e03d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:37.419828 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-6e985657-4554-4d15-b704-0e8e1472b0c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:37.421755 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 785/1589] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:24:37 2022] POST /identity/v3/projects => generated 373 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:24:37.426875 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea0138d-f123-4b5f-8081-ad20bac6f2f3 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:37.444512 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:37.510958 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:37.542015 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea0138d-f123-4b5f-8081-ad20bac6f2f3 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:37.549245 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a46feb99-c701-4adf-aaa5-61f4186530dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:37.550938 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46feb99-c701-4adf-aaa5-61f4186530dd admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:37.551376 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46feb99-c701-4adf-aaa5-61f4186530dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:37.551828 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46feb99-c701-4adf-aaa5-61f4186530dd admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:37.553352 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a46feb99-c701-4adf-aaa5-61f4186530dd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:37.556702 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a46feb99-c701-4adf-aaa5-61f4186530dd admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:37.581518 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 805/1590] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:37 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:37.595715 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 786/1591] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:24:37 2022] POST /identity/v3/users => generated 293 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:24:37.595892 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e985657-4554-4d15-b704-0e8e1472b0c8 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:37.599150 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-6e985657-4554-4d15-b704-0e8e1472b0c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:37.600317 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a46feb99-c701-4adf-aaa5-61f4186530dd admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:37.615698 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a46feb99-c701-4adf-aaa5-61f4186530dd admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:37.623496 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae29de3-9998-4b0a-8759-692cc2a1c4f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:37.624356 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae29de3-9998-4b0a-8759-692cc2a1c4f2 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:37.624810 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae29de3-9998-4b0a-8759-692cc2a1c4f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:37.625390 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae29de3-9998-4b0a-8759-692cc2a1c4f2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:37.626832 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eae29de3-9998-4b0a-8759-692cc2a1c4f2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:37.629925 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eae29de3-9998-4b0a-8759-692cc2a1c4f2 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:37.649435 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 787/1592] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:37 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:37.653312 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae29de3-9998-4b0a-8759-692cc2a1c4f2 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:37.670294 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae29de3-9998-4b0a-8759-692cc2a1c4f2 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:37.675733 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286b2c04-30b8-4f5c-9b70-cd488c6dbcda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:37.676469 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286b2c04-30b8-4f5c-9b70-cd488c6dbcda admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:37.676749 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286b2c04-30b8-4f5c-9b70-cd488c6dbcda admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:37.677021 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286b2c04-30b8-4f5c-9b70-cd488c6dbcda admin admin] PATH_INFO: `/v3/projects/dd063b242052473a81b534c9f89ac3e0/users/5dbce7d7cf0f44dfa38691c1e3c38b57/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:37.719537 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e985657-4554-4d15-b704-0e8e1472b0c8 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:37.722417 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286b2c04-30b8-4f5c-9b70-cd488c6dbcda admin admin] RBAC: Authorizing `identity:create_grant(project_id=dd063b242052473a81b534c9f89ac3e0, user_id=5dbce7d7cf0f44dfa38691c1e3c38b57, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:37.724835 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd934d6-2bbd-4f18-9b21-a1ff02f1c580 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:37.725353 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286b2c04-30b8-4f5c-9b70-cd488c6dbcda admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:37.726017 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd934d6-2bbd-4f18-9b21-a1ff02f1c580 service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:37.726445 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd934d6-2bbd-4f18-9b21-a1ff02f1c580 service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:37.726863 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd934d6-2bbd-4f18-9b21-a1ff02f1c580 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:37.768424 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 788/1593] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:24:37 2022] PUT /identity/v3/projects/dd063b242052473a81b534c9f89ac3e0/users/5dbce7d7cf0f44dfa38691c1e3c38b57/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:37.769266 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286b2c04-30b8-4f5c-9b70-cd488c6dbcda admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:37.772601 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-286b2c04-30b8-4f5c-9b70-cd488c6dbcda admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:37.819282 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd934d6-2bbd-4f18-9b21-a1ff02f1c580 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:37.822871 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd934d6-2bbd-4f18-9b21-a1ff02f1c580 service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:37.834688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 806/1594] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:37 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 240 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:24:37.835706 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd934d6-2bbd-4f18-9b21-a1ff02f1c580 service cinder] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:37.849157 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd934d6-2bbd-4f18-9b21-a1ff02f1c580 service cinder] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:37.878263 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286b2c04-30b8-4f5c-9b70-cd488c6dbcda admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:37.886352 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5aca671-37bd-4ae6-bd6a-e8b08eef8ef1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:37.886645 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5aca671-37bd-4ae6-bd6a-e8b08eef8ef1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:37.887082 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5aca671-37bd-4ae6-bd6a-e8b08eef8ef1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:37.887357 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5aca671-37bd-4ae6-bd6a-e8b08eef8ef1 admin admin] PATH_INFO: `/v3/projects/3990cfa81b9d46318a34bb862d243121` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:37.888610 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5aca671-37bd-4ae6-bd6a-e8b08eef8ef1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3990cfa81b9d46318a34bb862d243121)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:37.894342 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5aca671-37bd-4ae6-bd6a-e8b08eef8ef1 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:37.925783 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1bc4ac-d622-4c2a-815d-5eaf34fa008c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:37.926312 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1bc4ac-d622-4c2a-815d-5eaf34fa008c admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:37.926693 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1bc4ac-d622-4c2a-815d-5eaf34fa008c admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:37.927102 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1bc4ac-d622-4c2a-815d-5eaf34fa008c admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:37.927929 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef1bc4ac-d622-4c2a-815d-5eaf34fa008c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:37.930301 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef1bc4ac-d622-4c2a-815d-5eaf34fa008c admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:37.934178 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:37.950782 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 807/1595] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:37 2022] GET /identity/v3/roles => generated 1420 bytes in 115 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:37.953244 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1bc4ac-d622-4c2a-815d-5eaf34fa008c admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:37.972143 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1bc4ac-d622-4c2a-815d-5eaf34fa008c admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:38.003577 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d5aca671-37bd-4ae6-bd6a-e8b08eef8ef1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3990cfa81b9d46318a34bb862d243121', 'request_id': 'req-d5aca671-37bd-4ae6-bd6a-e8b08eef8ef1'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:38.018798 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d5aca671-37bd-4ae6-bd6a-e8b08eef8ef1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3990cfa81b9d46318a34bb862d243121', 'request_id': 'req-d5aca671-37bd-4ae6-bd6a-e8b08eef8ef1'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:38.033524 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d5aca671-37bd-4ae6-bd6a-e8b08eef8ef1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3990cfa81b9d46318a34bb862d243121', 'request_id': 'req-d5aca671-37bd-4ae6-bd6a-e8b08eef8ef1'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:38.074893 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d5aca671-37bd-4ae6-bd6a-e8b08eef8ef1 admin admin] The token cache is being invalidate because project 3990cfa81b9d46318a34bb862d243121 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:24:38.075248 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d5aca671-37bd-4ae6-bd6a-e8b08eef8ef1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:38.084656 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d5aca671-37bd-4ae6-bd6a-e8b08eef8ef1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:38.089555 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 789/1596] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:24:37 2022] DELETE /identity/v3/projects/3990cfa81b9d46318a34bb862d243121 => generated 0 bytes in 321 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:38.094415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5aca671-37bd-4ae6-bd6a-e8b08eef8ef1 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:38.096001 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d5aca671-37bd-4ae6-bd6a-e8b08eef8ef1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:38.113939 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2e9a9b-a8dd-49d2-8453-e30eb77dd09f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:38.116176 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2e9a9b-a8dd-49d2-8453-e30eb77dd09f admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:38.119298 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2e9a9b-a8dd-49d2-8453-e30eb77dd09f admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:38.120702 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2e9a9b-a8dd-49d2-8453-e30eb77dd09f admin admin] PATH_INFO: `/v3/projects/dd063b242052473a81b534c9f89ac3e0/users/5dbce7d7cf0f44dfa38691c1e3c38b57/roles/414286927fdd44688107ff5e10235acf` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:38.126764 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc2e9a9b-a8dd-49d2-8453-e30eb77dd09f admin admin] RBAC: Authorizing `identity:create_grant(project_id=dd063b242052473a81b534c9f89ac3e0, user_id=5dbce7d7cf0f44dfa38691c1e3c38b57, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:38.130368 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5aca671-37bd-4ae6-bd6a-e8b08eef8ef1 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:38.136357 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc2e9a9b-a8dd-49d2-8453-e30eb77dd09f admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:38.146661 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3b7de0-ae98-436e-9a62-3eb18d4fbdab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:38.148539 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3b7de0-ae98-436e-9a62-3eb18d4fbdab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:38.151428 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3b7de0-ae98-436e-9a62-3eb18d4fbdab admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:38.151787 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3b7de0-ae98-436e-9a62-3eb18d4fbdab admin admin] PATH_INFO: `/v3/users/83ead3aaced3494facc375590f6f68b8` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:38.153763 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3b7de0-ae98-436e-9a62-3eb18d4fbdab admin admin] RBAC: Authorizing `identity:delete_user(user_id=83ead3aaced3494facc375590f6f68b8)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:38.160731 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3b7de0-ae98-436e-9a62-3eb18d4fbdab admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:38.170442 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 808/1597] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:24:37 2022] PUT /identity/v3/projects/dd063b242052473a81b534c9f89ac3e0/users/5dbce7d7cf0f44dfa38691c1e3c38b57/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:38.182518 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14a64cc-8120-4dd6-bb86-e608496a8b66 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:38.182824 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14a64cc-8120-4dd6-bb86-e608496a8b66 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:38.183098 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14a64cc-8120-4dd6-bb86-e608496a8b66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:38.241098 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=85,keystone:INSERT=5,keystone:DELETE=10 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:38.273916 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-c14a64cc-8120-4dd6-bb86-e608496a8b66 None None] MFA Rules not processed for user `5dbce7d7cf0f44dfa38691c1e3c38b57`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:38.276170 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-c14a64cc-8120-4dd6-bb86-e608496a8b66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:38.288165 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-cb3b7de0-ae98-436e-9a62-3eb18d4fbdab admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '83ead3aaced3494facc375590f6f68b8'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:38.322402 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-cb3b7de0-ae98-436e-9a62-3eb18d4fbdab admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '83ead3aaced3494facc375590f6f68b8'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:38.324009 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-cb3b7de0-ae98-436e-9a62-3eb18d4fbdab admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '83ead3aaced3494facc375590f6f68b8'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:38.384021 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-cb3b7de0-ae98-436e-9a62-3eb18d4fbdab admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '83ead3aaced3494facc375590f6f68b8'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:38.407611 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-cb3b7de0-ae98-436e-9a62-3eb18d4fbdab admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '83ead3aaced3494facc375590f6f68b8'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:38.422700 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 790/1598] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:24:38 2022] DELETE /identity/v3/users/83ead3aaced3494facc375590f6f68b8 => generated 0 bytes in 327 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:38.522235 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 809/1599] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:38 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 338 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:24:38.530477 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3b7de0-ae98-436e-9a62-3eb18d4fbdab admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:38.533687 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-cb3b7de0-ae98-436e-9a62-3eb18d4fbdab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:38.638812 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3b7de0-ae98-436e-9a62-3eb18d4fbdab admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:38.644931 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa04b95-65eb-4591-9996-24ff541bb0dd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:38.645603 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa04b95-65eb-4591-9996-24ff541bb0dd service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:38.646209 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa04b95-65eb-4591-9996-24ff541bb0dd service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:38.646499 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa04b95-65eb-4591-9996-24ff541bb0dd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:38.661342 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613524b8-a9a7-4231-9547-943184a8dbb3 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:38.661739 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613524b8-a9a7-4231-9547-943184a8dbb3 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:38.662119 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613524b8-a9a7-4231-9547-943184a8dbb3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:38.697285 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-613524b8-a9a7-4231-9547-943184a8dbb3 None None] MFA Rules not processed for user `6bd65bfc7c994b58a47b2314bedd8c84`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:38.699114 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-613524b8-a9a7-4231-9547-943184a8dbb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:38.789286 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fa04b95-65eb-4591-9996-24ff541bb0dd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:38.790958 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fa04b95-65eb-4591-9996-24ff541bb0dd service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:38.849631 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 791/1600] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:38 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 27 01:24:38.870120 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 810/1601] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:38 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:38.885303 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa04b95-65eb-4591-9996-24ff541bb0dd service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:38.887110 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-1fa04b95-65eb-4591-9996-24ff541bb0dd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:38.914840 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa04b95-65eb-4591-9996-24ff541bb0dd service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:38.920921 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec9cf71e-101b-485b-b9a4-f4811651a53c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:38.921799 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec9cf71e-101b-485b-b9a4-f4811651a53c service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:38.922258 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec9cf71e-101b-485b-b9a4-f4811651a53c service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:38.922778 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec9cf71e-101b-485b-b9a4-f4811651a53c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:38.925796 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ec9cf71e-101b-485b-b9a4-f4811651a53c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:38.999823 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:DELETE=7,keystone:SELECT=92,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:39.026756 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec9cf71e-101b-485b-b9a4-f4811651a53c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:39.028849 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec9cf71e-101b-485b-b9a4-f4811651a53c service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:39.040883 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 792/1602] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:38 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:24:39.066336 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613524b8-a9a7-4231-9547-943184a8dbb3 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:39.068097 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-613524b8-a9a7-4231-9547-943184a8dbb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:39.076038 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:39.168739 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613524b8-a9a7-4231-9547-943184a8dbb3 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:39.178565 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-457c93fb-b665-46bf-b494-1376385cab67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:39.178565 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457c93fb-b665-46bf-b494-1376385cab67 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:39.178565 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457c93fb-b665-46bf-b494-1376385cab67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:39.178565 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457c93fb-b665-46bf-b494-1376385cab67 admin admin] PATH_INFO: `/v3/projects/f6ffdabbaf92472296a0dd36f6123ebb` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:39.179545 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-457c93fb-b665-46bf-b494-1376385cab67 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f6ffdabbaf92472296a0dd36f6123ebb)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:39.183810 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-457c93fb-b665-46bf-b494-1376385cab67 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:39.284851 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-457c93fb-b665-46bf-b494-1376385cab67 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f6ffdabbaf92472296a0dd36f6123ebb', 'request_id': 'req-457c93fb-b665-46bf-b494-1376385cab67'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:39.298453 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-457c93fb-b665-46bf-b494-1376385cab67 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f6ffdabbaf92472296a0dd36f6123ebb', 'request_id': 'req-457c93fb-b665-46bf-b494-1376385cab67'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:39.315451 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-457c93fb-b665-46bf-b494-1376385cab67 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f6ffdabbaf92472296a0dd36f6123ebb', 'request_id': 'req-457c93fb-b665-46bf-b494-1376385cab67'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:39.342909 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-457c93fb-b665-46bf-b494-1376385cab67 admin admin] The token cache is being invalidate because project f6ffdabbaf92472296a0dd36f6123ebb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:24:39.343210 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-457c93fb-b665-46bf-b494-1376385cab67 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:39.344095 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-457c93fb-b665-46bf-b494-1376385cab67 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:39.350901 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 811/1603] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:24:39 2022] DELETE /identity/v3/projects/f6ffdabbaf92472296a0dd36f6123ebb => generated 0 bytes in 281 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:39.351100 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec9cf71e-101b-485b-b9a4-f4811651a53c service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:39.367717 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec9cf71e-101b-485b-b9a4-f4811651a53c service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:39.438065 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-457c93fb-b665-46bf-b494-1376385cab67 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:39.440261 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-457c93fb-b665-46bf-b494-1376385cab67 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:39.494620 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4624aff-09fd-4a54-b8f7-1ededd2ee6ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:39.495409 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4624aff-09fd-4a54-b8f7-1ededd2ee6ef admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:39.495690 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4624aff-09fd-4a54-b8f7-1ededd2ee6ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:39.495958 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4624aff-09fd-4a54-b8f7-1ededd2ee6ef admin admin] PATH_INFO: `/v3/users/812f282a9ee64d55b1dc9523b030d53d` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:39.497392 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4624aff-09fd-4a54-b8f7-1ededd2ee6ef admin admin] RBAC: Authorizing `identity:delete_user(user_id=812f282a9ee64d55b1dc9523b030d53d)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:39.499324 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4624aff-09fd-4a54-b8f7-1ededd2ee6ef admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:39.598527 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b4624aff-09fd-4a54-b8f7-1ededd2ee6ef admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '812f282a9ee64d55b1dc9523b030d53d'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:39.609440 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b4624aff-09fd-4a54-b8f7-1ededd2ee6ef admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '812f282a9ee64d55b1dc9523b030d53d'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:39.610071 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b4624aff-09fd-4a54-b8f7-1ededd2ee6ef admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '812f282a9ee64d55b1dc9523b030d53d'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:39.615868 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-457c93fb-b665-46bf-b494-1376385cab67 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:39.624738 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21bf959c-7cdb-4601-8411-76a512dfad90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:39.625509 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bf959c-7cdb-4601-8411-76a512dfad90 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:39.625937 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bf959c-7cdb-4601-8411-76a512dfad90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:39.626227 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bf959c-7cdb-4601-8411-76a512dfad90 admin admin] PATH_INFO: `/v3/users/2b2f77ca967a48e1aec54a4771ec0df0` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:39.627681 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21bf959c-7cdb-4601-8411-76a512dfad90 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2b2f77ca967a48e1aec54a4771ec0df0)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:39.631971 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21bf959c-7cdb-4601-8411-76a512dfad90 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:39.636407 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b4624aff-09fd-4a54-b8f7-1ededd2ee6ef admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '812f282a9ee64d55b1dc9523b030d53d'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:39.655651 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b4624aff-09fd-4a54-b8f7-1ededd2ee6ef admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '812f282a9ee64d55b1dc9523b030d53d'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:39.664108 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 793/1604] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:24:39 2022] DELETE /identity/v3/users/812f282a9ee64d55b1dc9523b030d53d => generated 0 bytes in 314 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:24:39.700899 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:39.715804 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-21bf959c-7cdb-4601-8411-76a512dfad90 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2b2f77ca967a48e1aec54a4771ec0df0'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:39.727228 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-21bf959c-7cdb-4601-8411-76a512dfad90 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2b2f77ca967a48e1aec54a4771ec0df0'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:39.743279 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-21bf959c-7cdb-4601-8411-76a512dfad90 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2b2f77ca967a48e1aec54a4771ec0df0'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:39.744038 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-21bf959c-7cdb-4601-8411-76a512dfad90 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2b2f77ca967a48e1aec54a4771ec0df0'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:39.744680 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-21bf959c-7cdb-4601-8411-76a512dfad90 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2b2f77ca967a48e1aec54a4771ec0df0'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:39.765754 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52cc489-9d5a-4fbe-bb6c-b8b50459319e None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:39.766040 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52cc489-9d5a-4fbe-bb6c-b8b50459319e None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:39.766442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52cc489-9d5a-4fbe-bb6c-b8b50459319e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:39.769126 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 812/1605] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:24:39 2022] DELETE /identity/v3/users/2b2f77ca967a48e1aec54a4771ec0df0 => generated 0 bytes in 324 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:39.787726 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-e52cc489-9d5a-4fbe-bb6c-b8b50459319e None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:39.789216 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e52cc489-9d5a-4fbe-bb6c-b8b50459319e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:39.880597 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 794/1606] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:39.890298 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21bf959c-7cdb-4601-8411-76a512dfad90 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:39.892010 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-21bf959c-7cdb-4601-8411-76a512dfad90 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:39.954376 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21bf959c-7cdb-4601-8411-76a512dfad90 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:39.958524 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2adf6425-6714-47b0-a6a0-22f94e8c6ec7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:39.959157 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2adf6425-6714-47b0-a6a0-22f94e8c6ec7 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:39.959436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2adf6425-6714-47b0-a6a0-22f94e8c6ec7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:39.959704 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2adf6425-6714-47b0-a6a0-22f94e8c6ec7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:39.969261 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2adf6425-6714-47b0-a6a0-22f94e8c6ec7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:39.970746 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2adf6425-6714-47b0-a6a0-22f94e8c6ec7 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:39.981798 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 813/1607] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:24:40.125987 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52cc489-9d5a-4fbe-bb6c-b8b50459319e None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:40.127985 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e52cc489-9d5a-4fbe-bb6c-b8b50459319e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:40.140021 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52cc489-9d5a-4fbe-bb6c-b8b50459319e None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:40.145388 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e96ba44-833c-49aa-b7fd-a3248ef24da9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:40.146022 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e96ba44-833c-49aa-b7fd-a3248ef24da9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:40.146301 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e96ba44-833c-49aa-b7fd-a3248ef24da9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:40.146605 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e96ba44-833c-49aa-b7fd-a3248ef24da9 admin admin] PATH_INFO: `/v3/projects/8ad725c644134e899181ac2a445e8f28` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:40.147715 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e96ba44-833c-49aa-b7fd-a3248ef24da9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8ad725c644134e899181ac2a445e8f28)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:40.149708 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e96ba44-833c-49aa-b7fd-a3248ef24da9 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:40.151370 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:40.239431 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4e96ba44-833c-49aa-b7fd-a3248ef24da9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8ad725c644134e899181ac2a445e8f28', 'request_id': 'req-4e96ba44-833c-49aa-b7fd-a3248ef24da9'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:40.248129 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4e96ba44-833c-49aa-b7fd-a3248ef24da9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8ad725c644134e899181ac2a445e8f28', 'request_id': 'req-4e96ba44-833c-49aa-b7fd-a3248ef24da9'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:40.255177 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4e96ba44-833c-49aa-b7fd-a3248ef24da9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8ad725c644134e899181ac2a445e8f28', 'request_id': 'req-4e96ba44-833c-49aa-b7fd-a3248ef24da9'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:40.266436 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4e96ba44-833c-49aa-b7fd-a3248ef24da9 admin admin] The token cache is being invalidate because project 8ad725c644134e899181ac2a445e8f28 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:24:40.266832 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4e96ba44-833c-49aa-b7fd-a3248ef24da9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:40.267600 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4e96ba44-833c-49aa-b7fd-a3248ef24da9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:40.270069 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 795/1608] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:24:40 2022] DELETE /identity/v3/projects/8ad725c644134e899181ac2a445e8f28 => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:40.275012 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971662f9-cfaa-4850-8d38-835e8a10fe91 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:40.275469 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971662f9-cfaa-4850-8d38-835e8a10fe91 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:40.275600 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971662f9-cfaa-4850-8d38-835e8a10fe91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:40.297242 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-971662f9-cfaa-4850-8d38-835e8a10fe91 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:40.297708 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e96ba44-833c-49aa-b7fd-a3248ef24da9 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:40.298504 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-971662f9-cfaa-4850-8d38-835e8a10fe91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:40.299098 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-4e96ba44-833c-49aa-b7fd-a3248ef24da9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:40.370424 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 814/1609] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:40.375085 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e96ba44-833c-49aa-b7fd-a3248ef24da9 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:40.378411 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac34b405-9993-4771-87fe-4563e9fbb0e5 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:40.378599 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac34b405-9993-4771-87fe-4563e9fbb0e5 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:40.378907 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac34b405-9993-4771-87fe-4563e9fbb0e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:40.381611 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12843f4c-0017-48ff-8e3c-8a4f5e767cdf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:40.382251 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12843f4c-0017-48ff-8e3c-8a4f5e767cdf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:40.382527 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12843f4c-0017-48ff-8e3c-8a4f5e767cdf admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:40.382817 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12843f4c-0017-48ff-8e3c-8a4f5e767cdf admin admin] PATH_INFO: `/v3/projects/86efb04a71c7402d87d34ca096d8d0fa` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:40.383927 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12843f4c-0017-48ff-8e3c-8a4f5e767cdf admin admin] RBAC: Authorizing `identity:delete_project(project_id=86efb04a71c7402d87d34ca096d8d0fa)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:40.385929 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12843f4c-0017-48ff-8e3c-8a4f5e767cdf admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:40.400708 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-ac34b405-9993-4771-87fe-4563e9fbb0e5 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:40.402191 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-ac34b405-9993-4771-87fe-4563e9fbb0e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:40.411185 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 815/1610] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:24:40.415316 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac34b405-9993-4771-87fe-4563e9fbb0e5 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:40.425248 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac34b405-9993-4771-87fe-4563e9fbb0e5 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:40.440791 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-12843f4c-0017-48ff-8e3c-8a4f5e767cdf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '86efb04a71c7402d87d34ca096d8d0fa', 'request_id': 'req-12843f4c-0017-48ff-8e3c-8a4f5e767cdf'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:40.448156 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-12843f4c-0017-48ff-8e3c-8a4f5e767cdf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '86efb04a71c7402d87d34ca096d8d0fa', 'request_id': 'req-12843f4c-0017-48ff-8e3c-8a4f5e767cdf'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:40.453805 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-12843f4c-0017-48ff-8e3c-8a4f5e767cdf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '86efb04a71c7402d87d34ca096d8d0fa', 'request_id': 'req-12843f4c-0017-48ff-8e3c-8a4f5e767cdf'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:40.465543 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-12843f4c-0017-48ff-8e3c-8a4f5e767cdf admin admin] The token cache is being invalidate because project 86efb04a71c7402d87d34ca096d8d0fa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:24:40.465965 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-12843f4c-0017-48ff-8e3c-8a4f5e767cdf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:40.466734 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-12843f4c-0017-48ff-8e3c-8a4f5e767cdf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:40.469087 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 796/1611] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:24:40 2022] DELETE /identity/v3/projects/86efb04a71c7402d87d34ca096d8d0fa => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:24:40.473481 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12843f4c-0017-48ff-8e3c-8a4f5e767cdf admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:40.474943 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-12843f4c-0017-48ff-8e3c-8a4f5e767cdf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:40.477359 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:40.494598 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585038c9-2e3f-40b1-aaee-eb1e6dc7d2a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:40.495117 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585038c9-2e3f-40b1-aaee-eb1e6dc7d2a8 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:40.495504 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585038c9-2e3f-40b1-aaee-eb1e6dc7d2a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:40.495838 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585038c9-2e3f-40b1-aaee-eb1e6dc7d2a8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:40.496648 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-585038c9-2e3f-40b1-aaee-eb1e6dc7d2a8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:24:40.497016 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-585038c9-2e3f-40b1-aaee-eb1e6dc7d2a8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:40.498957 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:40.499736 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-585038c9-2e3f-40b1-aaee-eb1e6dc7d2a8 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:40.516999 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 816/1612] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:24:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:24:40.518676 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585038c9-2e3f-40b1-aaee-eb1e6dc7d2a8 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:40.521395 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-585038c9-2e3f-40b1-aaee-eb1e6dc7d2a8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:40.537944 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585038c9-2e3f-40b1-aaee-eb1e6dc7d2a8 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:40.544795 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6352a2-5e38-4fab-877c-e9ec08c58674 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:40.545716 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6352a2-5e38-4fab-877c-e9ec08c58674 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:40.546120 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6352a2-5e38-4fab-877c-e9ec08c58674 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:40.546533 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6352a2-5e38-4fab-877c-e9ec08c58674 admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:40.547222 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c6352a2-5e38-4fab-877c-e9ec08c58674 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:40.549133 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c6352a2-5e38-4fab-877c-e9ec08c58674 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:40.560134 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12843f4c-0017-48ff-8e3c-8a4f5e767cdf admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:40.567224 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d74ec477-e308-42e4-8e85-ce3ecea6e5d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:40.568103 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74ec477-e308-42e4-8e85-ce3ecea6e5d4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:40.568447 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74ec477-e308-42e4-8e85-ce3ecea6e5d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:40.568817 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74ec477-e308-42e4-8e85-ce3ecea6e5d4 admin admin] PATH_INFO: `/v3/users/5dbce7d7cf0f44dfa38691c1e3c38b57` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:40.570189 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d74ec477-e308-42e4-8e85-ce3ecea6e5d4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5dbce7d7cf0f44dfa38691c1e3c38b57)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:40.572273 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d74ec477-e308-42e4-8e85-ce3ecea6e5d4 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:40.606356 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 817/1613] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:24:40 2022] POST /identity/v3/projects => generated 385 bytes in 85 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:24:40.607207 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6352a2-5e38-4fab-877c-e9ec08c58674 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:40.636772 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d74ec477-e308-42e4-8e85-ce3ecea6e5d4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5dbce7d7cf0f44dfa38691c1e3c38b57'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:40.646937 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d74ec477-e308-42e4-8e85-ce3ecea6e5d4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5dbce7d7cf0f44dfa38691c1e3c38b57'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:40.649672 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d74ec477-e308-42e4-8e85-ce3ecea6e5d4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5dbce7d7cf0f44dfa38691c1e3c38b57'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:40.666318 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d74ec477-e308-42e4-8e85-ce3ecea6e5d4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5dbce7d7cf0f44dfa38691c1e3c38b57'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:40.676582 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d74ec477-e308-42e4-8e85-ce3ecea6e5d4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5dbce7d7cf0f44dfa38691c1e3c38b57'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:40.682349 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6352a2-5e38-4fab-877c-e9ec08c58674 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:40.684894 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 797/1614] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:24:40 2022] DELETE /identity/v3/users/5dbce7d7cf0f44dfa38691c1e3c38b57 => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:40.687888 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d74ec477-e308-42e4-8e85-ce3ecea6e5d4 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:40.688220 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d74ec477-e308-42e4-8e85-ce3ecea6e5d4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:40.696602 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d81d79-6823-4da6-9dea-bb0725bb669f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:40.698928 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d81d79-6823-4da6-9dea-bb0725bb669f admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:40.700539 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d81d79-6823-4da6-9dea-bb0725bb669f admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:40.701010 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d81d79-6823-4da6-9dea-bb0725bb669f admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:40.703232 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34d81d79-6823-4da6-9dea-bb0725bb669f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:40.706334 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34d81d79-6823-4da6-9dea-bb0725bb669f admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:40.762210 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 818/1615] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:24:40 2022] POST /identity/v3/users => generated 293 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:24:40.763818 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2db6c5-b490-4024-9831-cfb187d92c07 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:40.764123 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2db6c5-b490-4024-9831-cfb187d92c07 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:40.764396 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2db6c5-b490-4024-9831-cfb187d92c07 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:40.765988 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 819/1616] 10.222.0.42 () {54 vars in 921 bytes} [Thu Oct 27 01:24:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:24:40.767572 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2db6c5-b490-4024-9831-cfb187d92c07 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:40.770684 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-2c2db6c5-b490-4024-9831-cfb187d92c07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:40.785785 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d74ec477-e308-42e4-8e85-ce3ecea6e5d4 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:40.790589 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d74492-cb29-4a6a-b3c8-37fe2934301e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:40.791210 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d74492-cb29-4a6a-b3c8-37fe2934301e service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:40.791484 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d74492-cb29-4a6a-b3c8-37fe2934301e service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:40.791754 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d74492-cb29-4a6a-b3c8-37fe2934301e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:40.793904 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-69d74492-cb29-4a6a-b3c8-37fe2934301e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:40.896542 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:40.921682 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2db6c5-b490-4024-9831-cfb187d92c07 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:40.927957 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121c9068-a5e3-44cd-94d6-35d90f5c0e22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:40.928534 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d74492-cb29-4a6a-b3c8-37fe2934301e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:40.929168 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121c9068-a5e3-44cd-94d6-35d90f5c0e22 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:40.929513 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121c9068-a5e3-44cd-94d6-35d90f5c0e22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:40.929877 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121c9068-a5e3-44cd-94d6-35d90f5c0e22 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:40.930977 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121c9068-a5e3-44cd-94d6-35d90f5c0e22 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:40.931259 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d74492-cb29-4a6a-b3c8-37fe2934301e service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:40.932883 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121c9068-a5e3-44cd-94d6-35d90f5c0e22 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:40.945521 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 798/1617] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:40 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 260 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:24:40.947458 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28598b5-446a-42d8-8cc6-39b7b791469c None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:40.947843 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28598b5-446a-42d8-8cc6-39b7b791469c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:40.948344 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28598b5-446a-42d8-8cc6-39b7b791469c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:40.962231 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 820/1618] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:40 2022] GET /identity/v3/roles => generated 1420 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:40.966122 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121c9068-a5e3-44cd-94d6-35d90f5c0e22 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:40.995508 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121c9068-a5e3-44cd-94d6-35d90f5c0e22 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:41.006219 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb5b3101-19c4-40dc-abcd-cc8c8afc1f98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:41.010325 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5b3101-19c4-40dc-abcd-cc8c8afc1f98 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:41.012055 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5b3101-19c4-40dc-abcd-cc8c8afc1f98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:41.012177 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-d28598b5-446a-42d8-8cc6-39b7b791469c None None] MFA Rules not processed for user `51033d56f85f4282a49ded6338440677`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:41.013656 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d28598b5-446a-42d8-8cc6-39b7b791469c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:41.014292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5b3101-19c4-40dc-abcd-cc8c8afc1f98 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:41.016714 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb5b3101-19c4-40dc-abcd-cc8c8afc1f98 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:41.025699 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb5b3101-19c4-40dc-abcd-cc8c8afc1f98 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:41.046887 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 821/1619] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:40 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:41.060190 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb5b3101-19c4-40dc-abcd-cc8c8afc1f98 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:41.090136 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb5b3101-19c4-40dc-abcd-cc8c8afc1f98 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:41.095389 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f2ae07-bfaa-43fd-925e-6e704369a645 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:41.096767 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f2ae07-bfaa-43fd-925e-6e704369a645 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:41.097333 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f2ae07-bfaa-43fd-925e-6e704369a645 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:41.097333 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f2ae07-bfaa-43fd-925e-6e704369a645 admin admin] PATH_INFO: `/v3/projects/c31fdacca68b460da5cdccb3d70a4c41/users/01c8f62cf46347d286e0b24f3104f56a/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:41.119731 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63f2ae07-bfaa-43fd-925e-6e704369a645 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c31fdacca68b460da5cdccb3d70a4c41, user_id=01c8f62cf46347d286e0b24f3104f56a, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:41.122449 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63f2ae07-bfaa-43fd-925e-6e704369a645 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:41.153238 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 822/1620] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:24:41 2022] PUT /identity/v3/projects/c31fdacca68b460da5cdccb3d70a4c41/users/01c8f62cf46347d286e0b24f3104f56a/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:41.162014 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f2ae07-bfaa-43fd-925e-6e704369a645 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:41.168911 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 799/1621] 10.222.0.42 () {58 vars in 1018 bytes} [Thu Oct 27 01:24:40 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:41.170296 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-63f2ae07-bfaa-43fd-925e-6e704369a645 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:41.172123 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f33d3d7-cc5f-42df-a694-d6252d699c93 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:41.172938 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f33d3d7-cc5f-42df-a694-d6252d699c93 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:41.173287 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f33d3d7-cc5f-42df-a694-d6252d699c93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:41.191378 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:41.218263 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-7f33d3d7-cc5f-42df-a694-d6252d699c93 None None] MFA Rules not processed for user `01c8f62cf46347d286e0b24f3104f56a`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:41.219652 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-7f33d3d7-cc5f-42df-a694-d6252d699c93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:41.264573 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f2ae07-bfaa-43fd-925e-6e704369a645 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:41.270550 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef8128f9-b5fa-4a7f-a96b-61027acbfb99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:41.271189 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8128f9-b5fa-4a7f-a96b-61027acbfb99 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:41.271468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8128f9-b5fa-4a7f-a96b-61027acbfb99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:41.271743 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8128f9-b5fa-4a7f-a96b-61027acbfb99 admin admin] PATH_INFO: `/v3/projects/dd063b242052473a81b534c9f89ac3e0` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:41.272888 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef8128f9-b5fa-4a7f-a96b-61027acbfb99 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dd063b242052473a81b534c9f89ac3e0)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:41.274945 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef8128f9-b5fa-4a7f-a96b-61027acbfb99 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:41.319048 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 800/1622] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:41 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:41.320188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f33d3d7-cc5f-42df-a694-d6252d699c93 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:41.321564 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-7f33d3d7-cc5f-42df-a694-d6252d699c93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:41.342539 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-ef8128f9-b5fa-4a7f-a96b-61027acbfb99 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dd063b242052473a81b534c9f89ac3e0', 'request_id': 'req-ef8128f9-b5fa-4a7f-a96b-61027acbfb99'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:41.352919 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-ef8128f9-b5fa-4a7f-a96b-61027acbfb99 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dd063b242052473a81b534c9f89ac3e0', 'request_id': 'req-ef8128f9-b5fa-4a7f-a96b-61027acbfb99'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:41.361457 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-ef8128f9-b5fa-4a7f-a96b-61027acbfb99 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dd063b242052473a81b534c9f89ac3e0', 'request_id': 'req-ef8128f9-b5fa-4a7f-a96b-61027acbfb99'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:41.373045 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-ef8128f9-b5fa-4a7f-a96b-61027acbfb99 admin admin] The token cache is being invalidate because project dd063b242052473a81b534c9f89ac3e0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:24:41.373364 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-ef8128f9-b5fa-4a7f-a96b-61027acbfb99 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:41.374027 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-ef8128f9-b5fa-4a7f-a96b-61027acbfb99 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:41.375780 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 823/1623] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:24:41 2022] DELETE /identity/v3/projects/dd063b242052473a81b534c9f89ac3e0 => generated 0 bytes in 222 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:41.376425 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef8128f9-b5fa-4a7f-a96b-61027acbfb99 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:41.386195 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-ef8128f9-b5fa-4a7f-a96b-61027acbfb99 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:41.406843 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f33d3d7-cc5f-42df-a694-d6252d699c93 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:41.412372 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:41.456394 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef8128f9-b5fa-4a7f-a96b-61027acbfb99 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:41.462811 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15cd4df3-6f81-4f1b-b573-b7944cb4979b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:41.463567 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15cd4df3-6f81-4f1b-b573-b7944cb4979b service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:41.463849 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15cd4df3-6f81-4f1b-b573-b7944cb4979b service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:41.464114 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15cd4df3-6f81-4f1b-b573-b7944cb4979b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:41.466414 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-15cd4df3-6f81-4f1b-b573-b7944cb4979b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:41.481051 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e7bb58-8c5f-40a9-97af-cbb3686d96c2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:41.481722 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e7bb58-8c5f-40a9-97af-cbb3686d96c2 service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:41.482001 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e7bb58-8c5f-40a9-97af-cbb3686d96c2 service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:41.482273 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e7bb58-8c5f-40a9-97af-cbb3686d96c2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:41.484016 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-b0e7bb58-8c5f-40a9-97af-cbb3686d96c2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:41.503183 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e7bb58-8c5f-40a9-97af-cbb3686d96c2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:41.504833 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e7bb58-8c5f-40a9-97af-cbb3686d96c2 service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:41.516661 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 801/1624] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:41 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 197 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:24:41.518219 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a354c797-c7bb-483c-b7dc-196219cc90be None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:41.518546 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a354c797-c7bb-483c-b7dc-196219cc90be None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:41.518932 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a354c797-c7bb-483c-b7dc-196219cc90be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:41.535205 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:41.538594 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15cd4df3-6f81-4f1b-b573-b7944cb4979b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:41.540323 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15cd4df3-6f81-4f1b-b573-b7944cb4979b service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:41.543773 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-a354c797-c7bb-483c-b7dc-196219cc90be None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:41.545211 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-a354c797-c7bb-483c-b7dc-196219cc90be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:41.567699 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 824/1625] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:41 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 192 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:24:41.619447 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 802/1626] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:41.627335 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df30b9bc-59c5-422d-8e5c-cb4ef80b3c84 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:41.627904 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df30b9bc-59c5-422d-8e5c-cb4ef80b3c84 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:41.628111 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df30b9bc-59c5-422d-8e5c-cb4ef80b3c84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:41.648004 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-df30b9bc-59c5-422d-8e5c-cb4ef80b3c84 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:41.649240 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-df30b9bc-59c5-422d-8e5c-cb4ef80b3c84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:41.657393 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 825/1627] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:41.662792 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a354c797-c7bb-483c-b7dc-196219cc90be None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:41.673088 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a354c797-c7bb-483c-b7dc-196219cc90be None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:41.678145 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b792d5-48b6-4d20-814c-39f3f15a55a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:41.678753 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b792d5-48b6-4d20-814c-39f3f15a55a3 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:41.679026 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b792d5-48b6-4d20-814c-39f3f15a55a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:41.679299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b792d5-48b6-4d20-814c-39f3f15a55a3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:41.679949 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62b792d5-48b6-4d20-814c-39f3f15a55a3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:24:41.680256 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62b792d5-48b6-4d20-814c-39f3f15a55a3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:41.682157 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62b792d5-48b6-4d20-814c-39f3f15a55a3 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:41.693995 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 803/1628] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:24:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:24:41.698340 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df30b9bc-59c5-422d-8e5c-cb4ef80b3c84 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:41.708121 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df30b9bc-59c5-422d-8e5c-cb4ef80b3c84 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:41.713299 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb6f0792-08b6-4d59-8ea5-a9d74d33784c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:41.713905 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6f0792-08b6-4d59-8ea5-a9d74d33784c admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:41.714189 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6f0792-08b6-4d59-8ea5-a9d74d33784c admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:41.714493 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6f0792-08b6-4d59-8ea5-a9d74d33784c admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:41.715182 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb6f0792-08b6-4d59-8ea5-a9d74d33784c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:41.717177 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb6f0792-08b6-4d59-8ea5-a9d74d33784c admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:41.747444 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 826/1629] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:24:41 2022] POST /identity/v3/projects => generated 409 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:24:41.751667 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b792d5-48b6-4d20-814c-39f3f15a55a3 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:41.762648 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b792d5-48b6-4d20-814c-39f3f15a55a3 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:41.847182 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4933ceb9-d402-4b53-b597-e0ec9d1ce633 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:41.847927 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4933ceb9-d402-4b53-b597-e0ec9d1ce633 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:41.848229 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4933ceb9-d402-4b53-b597-e0ec9d1ce633 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:41.848500 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4933ceb9-d402-4b53-b597-e0ec9d1ce633 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:41.849217 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4933ceb9-d402-4b53-b597-e0ec9d1ce633 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:41.851163 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4933ceb9-d402-4b53-b597-e0ec9d1ce633 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:41.893633 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 804/1630] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:24:41 2022] POST /identity/v3/users => generated 305 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:24:41.897841 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb6f0792-08b6-4d59-8ea5-a9d74d33784c admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:41.909280 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb6f0792-08b6-4d59-8ea5-a9d74d33784c admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:41.916012 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffbbe6c3-9616-42c7-bf5e-56c83e0e925c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:41.916601 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffbbe6c3-9616-42c7-bf5e-56c83e0e925c admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:41.916888 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffbbe6c3-9616-42c7-bf5e-56c83e0e925c admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:41.917165 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffbbe6c3-9616-42c7-bf5e-56c83e0e925c admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:41.918120 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffbbe6c3-9616-42c7-bf5e-56c83e0e925c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:41.919694 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffbbe6c3-9616-42c7-bf5e-56c83e0e925c admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:41.931571 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 827/1631] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:41 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:41.935806 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4933ceb9-d402-4b53-b597-e0ec9d1ce633 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:41.946621 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4933ceb9-d402-4b53-b597-e0ec9d1ce633 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:41.951129 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7667eed4-478b-4c1c-ab07-a57004a87461 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:41.951711 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7667eed4-478b-4c1c-ab07-a57004a87461 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:41.951984 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7667eed4-478b-4c1c-ab07-a57004a87461 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:41.952249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7667eed4-478b-4c1c-ab07-a57004a87461 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:41.952872 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7667eed4-478b-4c1c-ab07-a57004a87461 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:41.954478 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7667eed4-478b-4c1c-ab07-a57004a87461 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:41.959576 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:41.965431 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 805/1632] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:41 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:41.969678 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffbbe6c3-9616-42c7-bf5e-56c83e0e925c admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:41.979231 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffbbe6c3-9616-42c7-bf5e-56c83e0e925c admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:41.983818 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67938ff7-3b88-4358-8e21-453ba25e5d84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:41.984470 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67938ff7-3b88-4358-8e21-453ba25e5d84 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:41.984751 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67938ff7-3b88-4358-8e21-453ba25e5d84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:41.985019 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67938ff7-3b88-4358-8e21-453ba25e5d84 admin admin] PATH_INFO: `/v3/projects/7bdee2fdcab94a87a832cc01c46361b2/users/b6aa96a98fe34a06a8f6151bf0f2ae9a/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:41.999932 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67938ff7-3b88-4358-8e21-453ba25e5d84 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7bdee2fdcab94a87a832cc01c46361b2, user_id=b6aa96a98fe34a06a8f6151bf0f2ae9a, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:42.003208 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67938ff7-3b88-4358-8e21-453ba25e5d84 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:42.015983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 828/1633] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:24:41 2022] PUT /identity/v3/projects/7bdee2fdcab94a87a832cc01c46361b2/users/b6aa96a98fe34a06a8f6151bf0f2ae9a/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:42.026332 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473d9e56-f2c9-4506-b35d-84983968419c None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:42.026622 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473d9e56-f2c9-4506-b35d-84983968419c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:42.026952 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473d9e56-f2c9-4506-b35d-84983968419c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:42.055375 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-473d9e56-f2c9-4506-b35d-84983968419c None None] MFA Rules not processed for user `b6aa96a98fe34a06a8f6151bf0f2ae9a`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:42.056855 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-473d9e56-f2c9-4506-b35d-84983968419c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:42.138543 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 806/1634] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:42 2022] POST /identity/v3/auth/tokens => generated 3286 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:24:42.143584 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67938ff7-3b88-4358-8e21-453ba25e5d84 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:42.212121 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67938ff7-3b88-4358-8e21-453ba25e5d84 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:42.222420 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31fa8479-23dd-4b09-a21e-3f92fe7bd6ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:42.223043 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fa8479-23dd-4b09-a21e-3f92fe7bd6ab admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:42.223322 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fa8479-23dd-4b09-a21e-3f92fe7bd6ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:42.223593 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fa8479-23dd-4b09-a21e-3f92fe7bd6ab admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:42.224251 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31fa8479-23dd-4b09-a21e-3f92fe7bd6ab admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:42.226184 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31fa8479-23dd-4b09-a21e-3f92fe7bd6ab admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:42.255246 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 829/1635] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:24:42 2022] POST /identity/v3/projects => generated 407 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:24:42.259553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473d9e56-f2c9-4506-b35d-84983968419c None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:42.270469 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473d9e56-f2c9-4506-b35d-84983968419c None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:42.327294 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afca5925-b5a4-4d67-a87c-3636212e7bd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:42.327895 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afca5925-b5a4-4d67-a87c-3636212e7bd3 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:42.328174 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afca5925-b5a4-4d67-a87c-3636212e7bd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:42.328709 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afca5925-b5a4-4d67-a87c-3636212e7bd3 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:42.329052 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afca5925-b5a4-4d67-a87c-3636212e7bd3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:42.330985 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afca5925-b5a4-4d67-a87c-3636212e7bd3 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:42.353165 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 807/1636] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:24:42 2022] POST /identity/v3/users => generated 310 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:24:42.357191 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31fa8479-23dd-4b09-a21e-3f92fe7bd6ab admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:42.368071 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31fa8479-23dd-4b09-a21e-3f92fe7bd6ab admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:42.372681 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce19f65-5917-4dc1-8e4d-8ac5a5252c88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:42.373391 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce19f65-5917-4dc1-8e4d-8ac5a5252c88 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:42.373666 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce19f65-5917-4dc1-8e4d-8ac5a5252c88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:42.373930 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce19f65-5917-4dc1-8e4d-8ac5a5252c88 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:42.374551 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce19f65-5917-4dc1-8e4d-8ac5a5252c88 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:42.376132 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce19f65-5917-4dc1-8e4d-8ac5a5252c88 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:42.386649 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 830/1637] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:42 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:42.390667 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afca5925-b5a4-4d67-a87c-3636212e7bd3 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:42.400767 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afca5925-b5a4-4d67-a87c-3636212e7bd3 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:42.405619 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af78cd60-fd41-4b3a-b996-1fbdfa6c603d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:42.406275 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af78cd60-fd41-4b3a-b996-1fbdfa6c603d admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:42.406557 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af78cd60-fd41-4b3a-b996-1fbdfa6c603d admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:42.406829 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af78cd60-fd41-4b3a-b996-1fbdfa6c603d admin admin] PATH_INFO: `/v3/projects/dfb2ecb7c2b347da96b157ded0d70c11/users/3d17102758cc40c8b50419f7912e023a/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:42.422659 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af78cd60-fd41-4b3a-b996-1fbdfa6c603d admin admin] RBAC: Authorizing `identity:create_grant(project_id=dfb2ecb7c2b347da96b157ded0d70c11, user_id=3d17102758cc40c8b50419f7912e023a, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:42.426014 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af78cd60-fd41-4b3a-b996-1fbdfa6c603d admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:42.441240 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 808/1638] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:24:42 2022] PUT /identity/v3/projects/dfb2ecb7c2b347da96b157ded0d70c11/users/3d17102758cc40c8b50419f7912e023a/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:42.445341 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce19f65-5917-4dc1-8e4d-8ac5a5252c88 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:42.455195 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce19f65-5917-4dc1-8e4d-8ac5a5252c88 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:42.514231 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7712a6-99f7-492f-85c8-7bbba66c006d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:42.514843 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7712a6-99f7-492f-85c8-7bbba66c006d admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:42.515124 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7712a6-99f7-492f-85c8-7bbba66c006d admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:42.515395 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7712a6-99f7-492f-85c8-7bbba66c006d admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:42.516059 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7712a6-99f7-492f-85c8-7bbba66c006d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:42.517708 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7712a6-99f7-492f-85c8-7bbba66c006d admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:42.528032 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 831/1639] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:42 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:42.532283 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af78cd60-fd41-4b3a-b996-1fbdfa6c603d admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:42.543102 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af78cd60-fd41-4b3a-b996-1fbdfa6c603d admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:42.547800 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3fbcdf-7fec-4927-9522-780f2753231c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:42.548488 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3fbcdf-7fec-4927-9522-780f2753231c admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:42.548767 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3fbcdf-7fec-4927-9522-780f2753231c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:42.549040 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3fbcdf-7fec-4927-9522-780f2753231c admin admin] PATH_INFO: `/v3/projects/dfb2ecb7c2b347da96b157ded0d70c11/users/3d17102758cc40c8b50419f7912e023a/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:42.551889 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd3fbcdf-7fec-4927-9522-780f2753231c admin admin] RBAC: Authorizing `identity:create_grant(project_id=dfb2ecb7c2b347da96b157ded0d70c11, user_id=3d17102758cc40c8b50419f7912e023a, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:42.554650 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd3fbcdf-7fec-4927-9522-780f2753231c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:42.565977 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 809/1640] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:24:42 2022] PUT /identity/v3/projects/dfb2ecb7c2b347da96b157ded0d70c11/users/3d17102758cc40c8b50419f7912e023a/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:42.577045 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3f4f24-bf52-4e4e-92c2-413f912d39d4 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:42.577388 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3f4f24-bf52-4e4e-92c2-413f912d39d4 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:42.577677 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3f4f24-bf52-4e4e-92c2-413f912d39d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:42.584672 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:42.608794 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-6c3f4f24-bf52-4e4e-92c2-413f912d39d4 None None] MFA Rules not processed for user `3d17102758cc40c8b50419f7912e023a`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:42.610035 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-6c3f4f24-bf52-4e4e-92c2-413f912d39d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:42.682289 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 832/1641] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:42 2022] POST /identity/v3/auth/tokens => generated 3351 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:42.693905 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3fbcdf-7fec-4927-9522-780f2753231c admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:42.743422 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:42.756097 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3fbcdf-7fec-4927-9522-780f2753231c admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:42.759724 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd6a000-d57f-4da4-bb6b-d3034628847d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:42.760364 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd6a000-d57f-4da4-bb6b-d3034628847d service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:42.760641 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd6a000-d57f-4da4-bb6b-d3034628847d service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:42.760910 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd6a000-d57f-4da4-bb6b-d3034628847d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:42.771083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd6a000-d57f-4da4-bb6b-d3034628847d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:42.772513 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd6a000-d57f-4da4-bb6b-d3034628847d service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:42.781702 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 810/1642] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:42 2022] GET /identity/v3/auth/tokens => generated 3351 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:24:42.944247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3f4f24-bf52-4e4e-92c2-413f912d39d4 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:42.955026 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3f4f24-bf52-4e4e-92c2-413f912d39d4 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:42.959203 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f9814c-e2ca-4cbc-9147-841de67eb91f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:42.959787 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f9814c-e2ca-4cbc-9147-841de67eb91f service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:42.960061 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f9814c-e2ca-4cbc-9147-841de67eb91f service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:42.960328 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f9814c-e2ca-4cbc-9147-841de67eb91f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:43.018902 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f9814c-e2ca-4cbc-9147-841de67eb91f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:43.020637 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f9814c-e2ca-4cbc-9147-841de67eb91f service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:43.030893 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 833/1643] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:42 2022] GET /identity/v3/auth/tokens => generated 3286 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:24:45.036998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd6a000-d57f-4da4-bb6b-d3034628847d service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:45.048526 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd6a000-d57f-4da4-bb6b-d3034628847d service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:45.055150 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61fdb9bd-2ec4-4bc5-bb0e-451ca7de98a6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:45.056004 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fdb9bd-2ec4-4bc5-bb0e-451ca7de98a6 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:45.056299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fdb9bd-2ec4-4bc5-bb0e-451ca7de98a6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:45.056569 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fdb9bd-2ec4-4bc5-bb0e-451ca7de98a6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:45.058582 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-61fdb9bd-2ec4-4bc5-bb0e-451ca7de98a6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:45.145991 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61fdb9bd-2ec4-4bc5-bb0e-451ca7de98a6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:45.147437 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61fdb9bd-2ec4-4bc5-bb0e-451ca7de98a6 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:45.157317 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 811/1644] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:45 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:24:53.028082 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=61 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:55.153426 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=37 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:56.833105 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f9814c-e2ca-4cbc-9147-841de67eb91f service cinder] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:56.834528 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-60f9814c-e2ca-4cbc-9147-841de67eb91f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:56.898413 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f9814c-e2ca-4cbc-9147-841de67eb91f service cinder] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:56.902881 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8abb54-16bf-4886-a13a-8370c2559366 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:56.903500 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8abb54-16bf-4886-a13a-8370c2559366 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:56.903778 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8abb54-16bf-4886-a13a-8370c2559366 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:56.904047 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8abb54-16bf-4886-a13a-8370c2559366 admin admin] PATH_INFO: `/v3/users/01c8f62cf46347d286e0b24f3104f56a` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:56.905182 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e8abb54-16bf-4886-a13a-8370c2559366 admin admin] RBAC: Authorizing `identity:delete_user(user_id=01c8f62cf46347d286e0b24f3104f56a)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:56.907098 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e8abb54-16bf-4886-a13a-8370c2559366 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:56.952483 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-6e8abb54-16bf-4886-a13a-8370c2559366 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '01c8f62cf46347d286e0b24f3104f56a'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:56.959184 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-6e8abb54-16bf-4886-a13a-8370c2559366 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '01c8f62cf46347d286e0b24f3104f56a'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:56.971006 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-6e8abb54-16bf-4886-a13a-8370c2559366 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '01c8f62cf46347d286e0b24f3104f56a'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:56.971698 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-6e8abb54-16bf-4886-a13a-8370c2559366 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '01c8f62cf46347d286e0b24f3104f56a'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:56.972334 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-6e8abb54-16bf-4886-a13a-8370c2559366 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '01c8f62cf46347d286e0b24f3104f56a'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:56.984931 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 834/1645] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:24:56 2022] DELETE /identity/v3/users/01c8f62cf46347d286e0b24f3104f56a => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:56.989755 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f082516e-745c-4274-8a68-de89f642852d None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:56.990221 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f082516e-745c-4274-8a68-de89f642852d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:56.990311 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f082516e-745c-4274-8a68-de89f642852d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:57.009644 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-f082516e-745c-4274-8a68-de89f642852d None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:57.010860 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f082516e-745c-4274-8a68-de89f642852d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:57.070749 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 812/1646] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:57.080439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8abb54-16bf-4886-a13a-8370c2559366 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:57.081917 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-6e8abb54-16bf-4886-a13a-8370c2559366 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:57.142655 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8abb54-16bf-4886-a13a-8370c2559366 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:57.146972 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd3bfc94-c41c-4ded-98ff-6c889d40922b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:57.147580 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3bfc94-c41c-4ded-98ff-6c889d40922b service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:57.147861 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3bfc94-c41c-4ded-98ff-6c889d40922b service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:57.148131 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3bfc94-c41c-4ded-98ff-6c889d40922b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:57.156911 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd3bfc94-c41c-4ded-98ff-6c889d40922b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:57.158370 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd3bfc94-c41c-4ded-98ff-6c889d40922b service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:57.167975 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 835/1647] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:24:57.464529 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f082516e-745c-4274-8a68-de89f642852d None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:57.469355 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f082516e-745c-4274-8a68-de89f642852d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:57.483024 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f082516e-745c-4274-8a68-de89f642852d None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:57.490118 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f33c7d-7c83-4c0f-9476-7381ad98820b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:57.491039 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f33c7d-7c83-4c0f-9476-7381ad98820b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:57.491320 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f33c7d-7c83-4c0f-9476-7381ad98820b admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:57.491941 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f33c7d-7c83-4c0f-9476-7381ad98820b admin admin] PATH_INFO: `/v3/projects/c31fdacca68b460da5cdccb3d70a4c41` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:57.493453 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f33c7d-7c83-4c0f-9476-7381ad98820b admin admin] RBAC: Authorizing `identity:delete_project(project_id=c31fdacca68b460da5cdccb3d70a4c41)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:57.495733 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f33c7d-7c83-4c0f-9476-7381ad98820b admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:57.560430 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b8f33c7d-7c83-4c0f-9476-7381ad98820b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c31fdacca68b460da5cdccb3d70a4c41', 'request_id': 'req-b8f33c7d-7c83-4c0f-9476-7381ad98820b'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:57.569288 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b8f33c7d-7c83-4c0f-9476-7381ad98820b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c31fdacca68b460da5cdccb3d70a4c41', 'request_id': 'req-b8f33c7d-7c83-4c0f-9476-7381ad98820b'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:57.576259 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b8f33c7d-7c83-4c0f-9476-7381ad98820b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c31fdacca68b460da5cdccb3d70a4c41', 'request_id': 'req-b8f33c7d-7c83-4c0f-9476-7381ad98820b'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:57.591132 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b8f33c7d-7c83-4c0f-9476-7381ad98820b admin admin] The token cache is being invalidate because project c31fdacca68b460da5cdccb3d70a4c41 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:24:57.591531 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b8f33c7d-7c83-4c0f-9476-7381ad98820b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:57.592658 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b8f33c7d-7c83-4c0f-9476-7381ad98820b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:24:57.598739 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 813/1648] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:24:57 2022] DELETE /identity/v3/projects/c31fdacca68b460da5cdccb3d70a4c41 => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:57.603571 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb63dc8-b590-4064-baa7-956c07a4243e None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:57.604107 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb63dc8-b590-4064-baa7-956c07a4243e None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:57.604444 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb63dc8-b590-4064-baa7-956c07a4243e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:57.626825 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-6cb63dc8-b590-4064-baa7-956c07a4243e None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:57.628033 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-6cb63dc8-b590-4064-baa7-956c07a4243e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:57.647181 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:57.698621 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 836/1649] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:57.706576 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f98e3e-0cda-49dd-8b90-b4909cc0f24e None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:57.707058 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f98e3e-0cda-49dd-8b90-b4909cc0f24e None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:57.707336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f98e3e-0cda-49dd-8b90-b4909cc0f24e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:57.746208 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-03f98e3e-0cda-49dd-8b90-b4909cc0f24e None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:57.747425 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-03f98e3e-0cda-49dd-8b90-b4909cc0f24e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:57.756473 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 814/1650] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:57.760531 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb63dc8-b590-4064-baa7-956c07a4243e None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:57.771287 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb63dc8-b590-4064-baa7-956c07a4243e None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:57.777227 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84037ea-4188-4224-9b9c-e75db28b8631 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:57.777915 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84037ea-4188-4224-9b9c-e75db28b8631 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:57.778363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84037ea-4188-4224-9b9c-e75db28b8631 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:57.778702 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84037ea-4188-4224-9b9c-e75db28b8631 admin admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:57.779467 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84037ea-4188-4224-9b9c-e75db28b8631 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:24:57.779839 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84037ea-4188-4224-9b9c-e75db28b8631 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:57.781536 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84037ea-4188-4224-9b9c-e75db28b8631 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:57.793657 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 837/1651] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:24:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:24:57.797947 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f98e3e-0cda-49dd-8b90-b4909cc0f24e None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:57.807658 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f98e3e-0cda-49dd-8b90-b4909cc0f24e None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:57.812376 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340df8a2-e23c-42ad-8fef-8690d96a1994 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:57.812974 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340df8a2-e23c-42ad-8fef-8690d96a1994 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:57.813278 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340df8a2-e23c-42ad-8fef-8690d96a1994 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:57.813545 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340df8a2-e23c-42ad-8fef-8690d96a1994 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:57.814171 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-340df8a2-e23c-42ad-8fef-8690d96a1994 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:57.816110 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-340df8a2-e23c-42ad-8fef-8690d96a1994 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:57.843074 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 815/1652] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:24:57 2022] POST /identity/v3/projects => generated 373 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:24:57.847310 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84037ea-4188-4224-9b9c-e75db28b8631 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:57.857722 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84037ea-4188-4224-9b9c-e75db28b8631 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:57.918481 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca93a020-6f19-4068-b592-700943b0d685 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:57.919086 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca93a020-6f19-4068-b592-700943b0d685 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:57.919366 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca93a020-6f19-4068-b592-700943b0d685 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:57.919639 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca93a020-6f19-4068-b592-700943b0d685 admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:57.920279 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca93a020-6f19-4068-b592-700943b0d685 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:57.922295 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca93a020-6f19-4068-b592-700943b0d685 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:57.943577 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 838/1653] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:24:57 2022] POST /identity/v3/users => generated 287 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:24:57.947795 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340df8a2-e23c-42ad-8fef-8690d96a1994 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:57.958365 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340df8a2-e23c-42ad-8fef-8690d96a1994 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:57.962902 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce310e8-209e-4d1c-9c29-02a77fc7777f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:57.963490 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce310e8-209e-4d1c-9c29-02a77fc7777f admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:57.963768 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce310e8-209e-4d1c-9c29-02a77fc7777f admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:57.964033 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce310e8-209e-4d1c-9c29-02a77fc7777f admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:57.964666 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce310e8-209e-4d1c-9c29-02a77fc7777f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:57.966299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce310e8-209e-4d1c-9c29-02a77fc7777f admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:57.977320 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 816/1654] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:57 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:57.981242 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca93a020-6f19-4068-b592-700943b0d685 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:57.991018 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca93a020-6f19-4068-b592-700943b0d685 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:57.995494 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d397fa7a-336e-4b28-8ea4-062150e1860b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:57.996081 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d397fa7a-336e-4b28-8ea4-062150e1860b admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:57.996359 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d397fa7a-336e-4b28-8ea4-062150e1860b admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:57.996628 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d397fa7a-336e-4b28-8ea4-062150e1860b admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:57.997302 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d397fa7a-336e-4b28-8ea4-062150e1860b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:57.998875 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d397fa7a-336e-4b28-8ea4-062150e1860b admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:58.009158 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 839/1655] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:24:57 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:24:58.013187 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce310e8-209e-4d1c-9c29-02a77fc7777f admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:58.023756 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce310e8-209e-4d1c-9c29-02a77fc7777f admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:58.028360 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d5f2805-9ac5-4f11-9f61-1148dee5ac4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:58.028997 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5f2805-9ac5-4f11-9f61-1148dee5ac4b admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:58.029307 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5f2805-9ac5-4f11-9f61-1148dee5ac4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:58.029578 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5f2805-9ac5-4f11-9f61-1148dee5ac4b admin admin] PATH_INFO: `/v3/projects/17f66c5afc1c4ee39262e5f1bcdbcaee/users/04eb00f1adb648f1a50d21a2003083a1/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:58.044701 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d5f2805-9ac5-4f11-9f61-1148dee5ac4b admin admin] RBAC: Authorizing `identity:create_grant(project_id=17f66c5afc1c4ee39262e5f1bcdbcaee, user_id=04eb00f1adb648f1a50d21a2003083a1, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:58.047445 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d5f2805-9ac5-4f11-9f61-1148dee5ac4b admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:58.060380 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 817/1656] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:24:58 2022] PUT /identity/v3/projects/17f66c5afc1c4ee39262e5f1bcdbcaee/users/04eb00f1adb648f1a50d21a2003083a1/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:24:58.070767 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e8b1ee-02ef-4b22-8088-b8a1983797a2 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:58.071046 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e8b1ee-02ef-4b22-8088-b8a1983797a2 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:58.071312 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e8b1ee-02ef-4b22-8088-b8a1983797a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:58.099420 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-11e8b1ee-02ef-4b22-8088-b8a1983797a2 None None] MFA Rules not processed for user `04eb00f1adb648f1a50d21a2003083a1`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:24:58.100527 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-11e8b1ee-02ef-4b22-8088-b8a1983797a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:58.164598 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 840/1657] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:24:58 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:24:58.175157 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d5f2805-9ac5-4f11-9f61-1148dee5ac4b admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:24:58.176884 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-4d5f2805-9ac5-4f11-9f61-1148dee5ac4b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:24:58.198190 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:24:58.235153 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d5f2805-9ac5-4f11-9f61-1148dee5ac4b admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:24:58.239152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af51d36e-9866-4ba3-8e65-f96f977fe802 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:24:58.239747 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af51d36e-9866-4ba3-8e65-f96f977fe802 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:24:58.240020 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af51d36e-9866-4ba3-8e65-f96f977fe802 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:24:58.240288 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af51d36e-9866-4ba3-8e65-f96f977fe802 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:24:58.248343 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af51d36e-9866-4ba3-8e65-f96f977fe802 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:24:58.249834 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af51d36e-9866-4ba3-8e65-f96f977fe802 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:24:58.258531 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 818/1658] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:24:58 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:25:03.205155 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e8b1ee-02ef-4b22-8088-b8a1983797a2 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:03.208924 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-11e8b1ee-02ef-4b22-8088-b8a1983797a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:03.243359 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:03.311446 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e8b1ee-02ef-4b22-8088-b8a1983797a2 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:03.317488 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa219d5-069e-4ebe-98ff-e03045ea2f59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:03.318150 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa219d5-069e-4ebe-98ff-e03045ea2f59 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:03.318427 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa219d5-069e-4ebe-98ff-e03045ea2f59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:03.318696 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa219d5-069e-4ebe-98ff-e03045ea2f59 admin admin] PATH_INFO: `/v3/users/b6aa96a98fe34a06a8f6151bf0f2ae9a` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:03.319843 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa219d5-069e-4ebe-98ff-e03045ea2f59 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b6aa96a98fe34a06a8f6151bf0f2ae9a)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:03.321880 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa219d5-069e-4ebe-98ff-e03045ea2f59 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:03.387868 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-8aa219d5-069e-4ebe-98ff-e03045ea2f59 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b6aa96a98fe34a06a8f6151bf0f2ae9a'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:03.398646 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-8aa219d5-069e-4ebe-98ff-e03045ea2f59 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b6aa96a98fe34a06a8f6151bf0f2ae9a'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:03.415460 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-8aa219d5-069e-4ebe-98ff-e03045ea2f59 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b6aa96a98fe34a06a8f6151bf0f2ae9a'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:03.416176 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-8aa219d5-069e-4ebe-98ff-e03045ea2f59 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b6aa96a98fe34a06a8f6151bf0f2ae9a'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:03.416854 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-8aa219d5-069e-4ebe-98ff-e03045ea2f59 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b6aa96a98fe34a06a8f6151bf0f2ae9a'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:03.431471 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 841/1659] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:25:03 2022] DELETE /identity/v3/users/b6aa96a98fe34a06a8f6151bf0f2ae9a => generated 0 bytes in 227 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:03.436810 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28dd9f83-b2f1-4ea4-b381-22449265771e None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:03.437443 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28dd9f83-b2f1-4ea4-b381-22449265771e None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:03.437443 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28dd9f83-b2f1-4ea4-b381-22449265771e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:03.459476 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-28dd9f83-b2f1-4ea4-b381-22449265771e None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:03.461568 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-28dd9f83-b2f1-4ea4-b381-22449265771e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:03.546541 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 819/1660] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:25:03.557602 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa219d5-069e-4ebe-98ff-e03045ea2f59 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:03.559001 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-8aa219d5-069e-4ebe-98ff-e03045ea2f59 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:03.616668 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa219d5-069e-4ebe-98ff-e03045ea2f59 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:03.620339 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3d48ca-caa2-4547-b303-a8d9c46b4cf9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:03.620935 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3d48ca-caa2-4547-b303-a8d9c46b4cf9 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:03.621244 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3d48ca-caa2-4547-b303-a8d9c46b4cf9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:03.621512 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3d48ca-caa2-4547-b303-a8d9c46b4cf9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:03.631062 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e3d48ca-caa2-4547-b303-a8d9c46b4cf9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:03.632470 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e3d48ca-caa2-4547-b303-a8d9c46b4cf9 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:03.641671 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 842/1661] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:25:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:25:04.107541 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28dd9f83-b2f1-4ea4-b381-22449265771e None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:04.109096 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-28dd9f83-b2f1-4ea4-b381-22449265771e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:04.122908 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28dd9f83-b2f1-4ea4-b381-22449265771e None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:04.127752 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04380f38-285d-4f43-8558-9dae0f3a8dc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:04.128536 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04380f38-285d-4f43-8558-9dae0f3a8dc4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:04.129566 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04380f38-285d-4f43-8558-9dae0f3a8dc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:04.129931 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04380f38-285d-4f43-8558-9dae0f3a8dc4 admin admin] PATH_INFO: `/v3/projects/7bdee2fdcab94a87a832cc01c46361b2` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:04.131079 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04380f38-285d-4f43-8558-9dae0f3a8dc4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7bdee2fdcab94a87a832cc01c46361b2)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:04.132989 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04380f38-285d-4f43-8558-9dae0f3a8dc4 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:04.185400 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-04380f38-285d-4f43-8558-9dae0f3a8dc4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7bdee2fdcab94a87a832cc01c46361b2', 'request_id': 'req-04380f38-285d-4f43-8558-9dae0f3a8dc4'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:04.194865 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-04380f38-285d-4f43-8558-9dae0f3a8dc4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7bdee2fdcab94a87a832cc01c46361b2', 'request_id': 'req-04380f38-285d-4f43-8558-9dae0f3a8dc4'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:04.201519 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-04380f38-285d-4f43-8558-9dae0f3a8dc4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7bdee2fdcab94a87a832cc01c46361b2', 'request_id': 'req-04380f38-285d-4f43-8558-9dae0f3a8dc4'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:04.213919 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-04380f38-285d-4f43-8558-9dae0f3a8dc4 admin admin] The token cache is being invalidate because project 7bdee2fdcab94a87a832cc01c46361b2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:25:04.214302 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-04380f38-285d-4f43-8558-9dae0f3a8dc4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:04.215019 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-04380f38-285d-4f43-8558-9dae0f3a8dc4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:04.216574 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 820/1662] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:25:04 2022] DELETE /identity/v3/projects/7bdee2fdcab94a87a832cc01c46361b2 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:04.220661 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3d48ca-caa2-4547-b303-a8d9c46b4cf9 service neutron] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:04.233583 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3d48ca-caa2-4547-b303-a8d9c46b4cf9 service neutron] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:04.275940 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:04.295778 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7050fc44-a327-4768-b38d-ea1200752d2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:04.296442 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7050fc44-a327-4768-b38d-ea1200752d2d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:04.296722 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7050fc44-a327-4768-b38d-ea1200752d2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:04.296994 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7050fc44-a327-4768-b38d-ea1200752d2d admin admin] PATH_INFO: `/v3/users/3d17102758cc40c8b50419f7912e023a` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:04.298144 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7050fc44-a327-4768-b38d-ea1200752d2d admin admin] RBAC: Authorizing `identity:delete_user(user_id=3d17102758cc40c8b50419f7912e023a)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:04.300059 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7050fc44-a327-4768-b38d-ea1200752d2d admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:04.346885 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-7050fc44-a327-4768-b38d-ea1200752d2d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3d17102758cc40c8b50419f7912e023a'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:04.353674 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-7050fc44-a327-4768-b38d-ea1200752d2d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3d17102758cc40c8b50419f7912e023a'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:04.366031 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-7050fc44-a327-4768-b38d-ea1200752d2d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3d17102758cc40c8b50419f7912e023a'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:04.366714 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-7050fc44-a327-4768-b38d-ea1200752d2d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3d17102758cc40c8b50419f7912e023a'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:04.367322 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-7050fc44-a327-4768-b38d-ea1200752d2d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3d17102758cc40c8b50419f7912e023a'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:04.381193 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 843/1663] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:25:04 2022] DELETE /identity/v3/users/3d17102758cc40c8b50419f7912e023a => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:04.650203 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04380f38-285d-4f43-8558-9dae0f3a8dc4 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:04.652894 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-04380f38-285d-4f43-8558-9dae0f3a8dc4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:04.668534 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04380f38-285d-4f43-8558-9dae0f3a8dc4 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:04.714243 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:04.728372 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e13acd5-d957-4d3c-8143-cee5832a7fd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:04.729019 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e13acd5-d957-4d3c-8143-cee5832a7fd7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:04.729319 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e13acd5-d957-4d3c-8143-cee5832a7fd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:04.729586 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e13acd5-d957-4d3c-8143-cee5832a7fd7 admin admin] PATH_INFO: `/v3/projects/dfb2ecb7c2b347da96b157ded0d70c11` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:04.730709 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e13acd5-d957-4d3c-8143-cee5832a7fd7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dfb2ecb7c2b347da96b157ded0d70c11)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:04.732637 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e13acd5-d957-4d3c-8143-cee5832a7fd7 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:04.779474 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-0e13acd5-d957-4d3c-8143-cee5832a7fd7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dfb2ecb7c2b347da96b157ded0d70c11', 'request_id': 'req-0e13acd5-d957-4d3c-8143-cee5832a7fd7'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:04.786569 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-0e13acd5-d957-4d3c-8143-cee5832a7fd7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dfb2ecb7c2b347da96b157ded0d70c11', 'request_id': 'req-0e13acd5-d957-4d3c-8143-cee5832a7fd7'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:04.791987 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-0e13acd5-d957-4d3c-8143-cee5832a7fd7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dfb2ecb7c2b347da96b157ded0d70c11', 'request_id': 'req-0e13acd5-d957-4d3c-8143-cee5832a7fd7'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:04.803815 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-0e13acd5-d957-4d3c-8143-cee5832a7fd7 admin admin] The token cache is being invalidate because project dfb2ecb7c2b347da96b157ded0d70c11 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:25:04.804100 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-0e13acd5-d957-4d3c-8143-cee5832a7fd7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:04.804803 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-0e13acd5-d957-4d3c-8143-cee5832a7fd7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:04.806773 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 821/1664] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:25:04 2022] DELETE /identity/v3/projects/dfb2ecb7c2b347da96b157ded0d70c11 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:04.812177 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd8c472-8647-440e-95d0-58cf31eafb98 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:04.812460 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd8c472-8647-440e-95d0-58cf31eafb98 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:04.812735 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd8c472-8647-440e-95d0-58cf31eafb98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:04.832528 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-afd8c472-8647-440e-95d0-58cf31eafb98 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:04.833752 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-afd8c472-8647-440e-95d0-58cf31eafb98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:04.891185 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 844/1665] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:25:04.898902 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a6de0e-9fab-484e-8580-3776c165e48b None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:04.899213 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a6de0e-9fab-484e-8580-3776c165e48b None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:04.899485 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a6de0e-9fab-484e-8580-3776c165e48b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:04.918903 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-d1a6de0e-9fab-484e-8580-3776c165e48b None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:04.920099 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d1a6de0e-9fab-484e-8580-3776c165e48b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:04.928494 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 822/1666] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:25:04.932806 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd8c472-8647-440e-95d0-58cf31eafb98 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:04.943107 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd8c472-8647-440e-95d0-58cf31eafb98 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:04.947677 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a3db66-bd38-4719-9db9-d607d322bc98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:04.948267 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a3db66-bd38-4719-9db9-d607d322bc98 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:04.948547 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a3db66-bd38-4719-9db9-d607d322bc98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:04.948818 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a3db66-bd38-4719-9db9-d607d322bc98 admin admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:04.949489 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a3db66-bd38-4719-9db9-d607d322bc98 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:25:04.949789 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a3db66-bd38-4719-9db9-d607d322bc98 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:04.951388 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a3db66-bd38-4719-9db9-d607d322bc98 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:04.962734 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 845/1667] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:25:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:25:04.966939 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a6de0e-9fab-484e-8580-3776c165e48b None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:04.977024 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a6de0e-9fab-484e-8580-3776c165e48b None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:04.981592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6f1e2d-e5fc-4978-9162-bdef17f1a324 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:04.982177 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6f1e2d-e5fc-4978-9162-bdef17f1a324 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:04.982451 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6f1e2d-e5fc-4978-9162-bdef17f1a324 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:04.982721 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6f1e2d-e5fc-4978-9162-bdef17f1a324 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:04.983340 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b6f1e2d-e5fc-4978-9162-bdef17f1a324 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:04.985290 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b6f1e2d-e5fc-4978-9162-bdef17f1a324 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:05.012473 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 823/1668] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:25:04 2022] POST /identity/v3/projects => generated 377 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:25:05.016586 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a3db66-bd38-4719-9db9-d607d322bc98 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:05.026610 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a3db66-bd38-4719-9db9-d607d322bc98 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:05.073713 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:05.091214 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84015fbb-e173-4680-88ac-b3fff2aeed88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:05.092117 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84015fbb-e173-4680-88ac-b3fff2aeed88 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:05.092755 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84015fbb-e173-4680-88ac-b3fff2aeed88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:05.093459 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84015fbb-e173-4680-88ac-b3fff2aeed88 admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:05.094649 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84015fbb-e173-4680-88ac-b3fff2aeed88 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:05.096810 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84015fbb-e173-4680-88ac-b3fff2aeed88 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:05.122311 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 846/1669] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:25:05 2022] POST /identity/v3/users => generated 289 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:25:05.126539 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6f1e2d-e5fc-4978-9162-bdef17f1a324 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:05.137925 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6f1e2d-e5fc-4978-9162-bdef17f1a324 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:05.142595 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8cfacd-af10-4960-82fa-0414b3597818 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:05.143205 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8cfacd-af10-4960-82fa-0414b3597818 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:05.143485 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8cfacd-af10-4960-82fa-0414b3597818 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:05.143758 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8cfacd-af10-4960-82fa-0414b3597818 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:05.144943 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8cfacd-af10-4960-82fa-0414b3597818 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:05.146668 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8cfacd-af10-4960-82fa-0414b3597818 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:05.158392 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 824/1670] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:25:05 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:25:05.163149 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84015fbb-e173-4680-88ac-b3fff2aeed88 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:05.173441 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84015fbb-e173-4680-88ac-b3fff2aeed88 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:05.178737 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6803a7c-377c-420e-aa39-a9ed2c8abe25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:05.179500 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6803a7c-377c-420e-aa39-a9ed2c8abe25 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:05.179763 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6803a7c-377c-420e-aa39-a9ed2c8abe25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:05.180032 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6803a7c-377c-420e-aa39-a9ed2c8abe25 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:05.181101 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6803a7c-377c-420e-aa39-a9ed2c8abe25 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:05.182644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6803a7c-377c-420e-aa39-a9ed2c8abe25 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:05.194249 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 847/1671] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:25:05 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:25:05.198439 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8cfacd-af10-4960-82fa-0414b3597818 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:05.208475 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8cfacd-af10-4960-82fa-0414b3597818 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:05.213907 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde96435-7e1d-4a7c-88fa-48220f43160f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:05.214659 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde96435-7e1d-4a7c-88fa-48220f43160f admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:05.214964 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde96435-7e1d-4a7c-88fa-48220f43160f admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:05.215235 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde96435-7e1d-4a7c-88fa-48220f43160f admin admin] PATH_INFO: `/v3/projects/218f9a606c4f493eb8eec0347f706043/users/d411dccf786346b1abfa3b2d3f9736b0/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:05.230871 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dde96435-7e1d-4a7c-88fa-48220f43160f admin admin] RBAC: Authorizing `identity:create_grant(project_id=218f9a606c4f493eb8eec0347f706043, user_id=d411dccf786346b1abfa3b2d3f9736b0, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:05.233567 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dde96435-7e1d-4a7c-88fa-48220f43160f admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:05.247538 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 825/1672] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:25:05 2022] PUT /identity/v3/projects/218f9a606c4f493eb8eec0347f706043/users/d411dccf786346b1abfa3b2d3f9736b0/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:05.258049 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe0f3b0-2e1d-497e-a014-a0e04cb31896 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:05.258497 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe0f3b0-2e1d-497e-a014-a0e04cb31896 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:05.258586 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe0f3b0-2e1d-497e-a014-a0e04cb31896 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:05.294038 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-7fe0f3b0-2e1d-497e-a014-a0e04cb31896 None None] MFA Rules not processed for user `d411dccf786346b1abfa3b2d3f9736b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:05.295448 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-7fe0f3b0-2e1d-497e-a014-a0e04cb31896 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:05.370390 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 848/1673] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:05 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:25:05.381384 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde96435-7e1d-4a7c-88fa-48220f43160f admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:05.383113 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-dde96435-7e1d-4a7c-88fa-48220f43160f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:05.477335 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde96435-7e1d-4a7c-88fa-48220f43160f admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:05.481724 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6c5a1b-84f9-4cde-8742-5679bf4e2a55 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2880dbb00eec471285d242e8183dd8d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:05.482519 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6c5a1b-84f9-4cde-8742-5679bf4e2a55 service glance] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:05.482953 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6c5a1b-84f9-4cde-8742-5679bf4e2a55 service glance] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:05.483368 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6c5a1b-84f9-4cde-8742-5679bf4e2a55 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:05.490740 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:05.493870 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c6c5a1b-84f9-4cde-8742-5679bf4e2a55 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:05.495897 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c6c5a1b-84f9-4cde-8742-5679bf4e2a55 service glance] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:05.507201 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 826/1674] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:25:05 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:25:15.366708 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=45,keystone:INSERT=3 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:15.509177 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:21.571023 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe0f3b0-2e1d-497e-a014-a0e04cb31896 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:21.572733 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-7fe0f3b0-2e1d-497e-a014-a0e04cb31896 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:21.699410 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe0f3b0-2e1d-497e-a014-a0e04cb31896 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:21.701294 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2eabb2-c40e-4d0f-9c4b-ec410099fc54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:21.702091 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2eabb2-c40e-4d0f-9c4b-ec410099fc54 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:21.703041 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2eabb2-c40e-4d0f-9c4b-ec410099fc54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:21.703310 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2eabb2-c40e-4d0f-9c4b-ec410099fc54 admin admin] PATH_INFO: `/v3/users/04eb00f1adb648f1a50d21a2003083a1` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:21.705301 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce2eabb2-c40e-4d0f-9c4b-ec410099fc54 admin admin] RBAC: Authorizing `identity:delete_user(user_id=04eb00f1adb648f1a50d21a2003083a1)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:21.706504 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce2eabb2-c40e-4d0f-9c4b-ec410099fc54 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:21.769839 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-ce2eabb2-c40e-4d0f-9c4b-ec410099fc54 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '04eb00f1adb648f1a50d21a2003083a1'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:21.779985 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-ce2eabb2-c40e-4d0f-9c4b-ec410099fc54 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '04eb00f1adb648f1a50d21a2003083a1'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:21.792773 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-ce2eabb2-c40e-4d0f-9c4b-ec410099fc54 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '04eb00f1adb648f1a50d21a2003083a1'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:21.793683 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-ce2eabb2-c40e-4d0f-9c4b-ec410099fc54 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '04eb00f1adb648f1a50d21a2003083a1'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:21.794275 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-ce2eabb2-c40e-4d0f-9c4b-ec410099fc54 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '04eb00f1adb648f1a50d21a2003083a1'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:21.808574 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 849/1675] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:25:21 2022] DELETE /identity/v3/users/04eb00f1adb648f1a50d21a2003083a1 => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:21.813280 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6532fdd-91e9-4d61-a8ec-1209b015bd45 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:21.813741 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6532fdd-91e9-4d61-a8ec-1209b015bd45 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:21.813817 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6532fdd-91e9-4d61-a8ec-1209b015bd45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:21.838514 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-a6532fdd-91e9-4d61-a8ec-1209b015bd45 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:21.840228 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-a6532fdd-91e9-4d61-a8ec-1209b015bd45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:21.928163 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 827/1676] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:25:21.937962 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2eabb2-c40e-4d0f-9c4b-ec410099fc54 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:21.939383 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-ce2eabb2-c40e-4d0f-9c4b-ec410099fc54 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:21.999618 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6532fdd-91e9-4d61-a8ec-1209b015bd45 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:21.999804 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2eabb2-c40e-4d0f-9c4b-ec410099fc54 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:22.001132 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-a6532fdd-91e9-4d61-a8ec-1209b015bd45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:22.004394 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa3bc0de-235e-4920-90c9-9bcf1743e91e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:22.005402 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3bc0de-235e-4920-90c9-9bcf1743e91e service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:22.005686 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3bc0de-235e-4920-90c9-9bcf1743e91e service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:22.005951 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3bc0de-235e-4920-90c9-9bcf1743e91e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:22.013412 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6532fdd-91e9-4d61-a8ec-1209b015bd45 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:22.015155 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa3bc0de-235e-4920-90c9-9bcf1743e91e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:22.016575 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa3bc0de-235e-4920-90c9-9bcf1743e91e service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:22.018922 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d030cd65-d900-4edb-9c4c-a7e7538fe56f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:22.019613 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d030cd65-d900-4edb-9c4c-a7e7538fe56f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:22.020034 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d030cd65-d900-4edb-9c4c-a7e7538fe56f admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:22.020387 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d030cd65-d900-4edb-9c4c-a7e7538fe56f admin admin] PATH_INFO: `/v3/users/0960e7d937cb4157a81d288e98d81f8f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:22.022675 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d030cd65-d900-4edb-9c4c-a7e7538fe56f admin admin] RBAC: Authorizing `identity:delete_user(user_id=0960e7d937cb4157a81d288e98d81f8f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:22.024890 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d030cd65-d900-4edb-9c4c-a7e7538fe56f admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:22.026521 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 850/1677] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:25:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:25:22.077403 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d030cd65-d900-4edb-9c4c-a7e7538fe56f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0960e7d937cb4157a81d288e98d81f8f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:22.085875 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d030cd65-d900-4edb-9c4c-a7e7538fe56f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0960e7d937cb4157a81d288e98d81f8f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:22.086512 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d030cd65-d900-4edb-9c4c-a7e7538fe56f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0960e7d937cb4157a81d288e98d81f8f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:22.101391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d030cd65-d900-4edb-9c4c-a7e7538fe56f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0960e7d937cb4157a81d288e98d81f8f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:22.111923 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d030cd65-d900-4edb-9c4c-a7e7538fe56f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0960e7d937cb4157a81d288e98d81f8f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:22.119013 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 828/1678] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:25:21 2022] DELETE /identity/v3/users/0960e7d937cb4157a81d288e98d81f8f => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:22.123885 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfc5861-2ff1-4ff3-81df-933c02beb79d None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:22.124271 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfc5861-2ff1-4ff3-81df-933c02beb79d None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:22.124744 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfc5861-2ff1-4ff3-81df-933c02beb79d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:22.145625 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-dbfc5861-2ff1-4ff3-81df-933c02beb79d None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:22.146818 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-dbfc5861-2ff1-4ff3-81df-933c02beb79d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:22.169864 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:22.259566 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 851/1679] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:25:22.265924 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d030cd65-d900-4edb-9c4c-a7e7538fe56f admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:22.267493 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d030cd65-d900-4edb-9c4c-a7e7538fe56f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:22.357371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d030cd65-d900-4edb-9c4c-a7e7538fe56f admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:22.369659 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084e9510-7440-4437-9635-362d2000e2f9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:22.370762 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084e9510-7440-4437-9635-362d2000e2f9 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:22.371181 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084e9510-7440-4437-9635-362d2000e2f9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:22.371451 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084e9510-7440-4437-9635-362d2000e2f9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:22.383382 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084e9510-7440-4437-9635-362d2000e2f9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:22.386729 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084e9510-7440-4437-9635-362d2000e2f9 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:22.402449 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 829/1680] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:25:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 27 01:25:22.454377 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbfc5861-2ff1-4ff3-81df-933c02beb79d None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:22.456243 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-dbfc5861-2ff1-4ff3-81df-933c02beb79d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:22.482684 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbfc5861-2ff1-4ff3-81df-933c02beb79d None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:22.488546 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f3998c5-3b6d-4dbc-994b-c0fd1cba8aaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:22.489892 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3998c5-3b6d-4dbc-994b-c0fd1cba8aaf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:22.490280 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3998c5-3b6d-4dbc-994b-c0fd1cba8aaf admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:22.490649 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3998c5-3b6d-4dbc-994b-c0fd1cba8aaf admin admin] PATH_INFO: `/v3/projects/17f66c5afc1c4ee39262e5f1bcdbcaee` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:22.491920 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f3998c5-3b6d-4dbc-994b-c0fd1cba8aaf admin admin] RBAC: Authorizing `identity:delete_project(project_id=17f66c5afc1c4ee39262e5f1bcdbcaee)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:22.494177 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f3998c5-3b6d-4dbc-994b-c0fd1cba8aaf admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:22.577752 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-8f3998c5-3b6d-4dbc-994b-c0fd1cba8aaf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '17f66c5afc1c4ee39262e5f1bcdbcaee', 'request_id': 'req-8f3998c5-3b6d-4dbc-994b-c0fd1cba8aaf'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:22.587504 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-8f3998c5-3b6d-4dbc-994b-c0fd1cba8aaf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '17f66c5afc1c4ee39262e5f1bcdbcaee', 'request_id': 'req-8f3998c5-3b6d-4dbc-994b-c0fd1cba8aaf'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:22.595347 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-8f3998c5-3b6d-4dbc-994b-c0fd1cba8aaf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '17f66c5afc1c4ee39262e5f1bcdbcaee', 'request_id': 'req-8f3998c5-3b6d-4dbc-994b-c0fd1cba8aaf'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:22.607585 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-8f3998c5-3b6d-4dbc-994b-c0fd1cba8aaf admin admin] The token cache is being invalidate because project 17f66c5afc1c4ee39262e5f1bcdbcaee was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:25:22.607829 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-8f3998c5-3b6d-4dbc-994b-c0fd1cba8aaf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:22.608499 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-8f3998c5-3b6d-4dbc-994b-c0fd1cba8aaf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:22.610312 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 852/1681] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:25:22 2022] DELETE /identity/v3/projects/17f66c5afc1c4ee39262e5f1bcdbcaee => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:22.616187 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab2b7fa-07b8-427c-b0e1-7cb089a43ecb None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:22.616473 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab2b7fa-07b8-427c-b0e1-7cb089a43ecb None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:22.616743 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab2b7fa-07b8-427c-b0e1-7cb089a43ecb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:22.639410 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-fab2b7fa-07b8-427c-b0e1-7cb089a43ecb None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:22.641092 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-fab2b7fa-07b8-427c-b0e1-7cb089a43ecb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:22.672165 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:22.760966 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 830/1682] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:25:22.768367 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48bc5a56-260b-4c19-b41d-7759f2e0c1a1 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:22.768743 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48bc5a56-260b-4c19-b41d-7759f2e0c1a1 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:22.769121 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48bc5a56-260b-4c19-b41d-7759f2e0c1a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:22.803311 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-48bc5a56-260b-4c19-b41d-7759f2e0c1a1 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:22.804531 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-48bc5a56-260b-4c19-b41d-7759f2e0c1a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:22.814409 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 853/1683] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:25:22.820171 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab2b7fa-07b8-427c-b0e1-7cb089a43ecb None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:22.836805 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab2b7fa-07b8-427c-b0e1-7cb089a43ecb None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:22.843272 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89aa1057-9378-4371-bd8f-70174b2cffee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:22.844328 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89aa1057-9378-4371-bd8f-70174b2cffee admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:22.844812 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89aa1057-9378-4371-bd8f-70174b2cffee admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:22.845546 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89aa1057-9378-4371-bd8f-70174b2cffee admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:22.846468 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89aa1057-9378-4371-bd8f-70174b2cffee admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:25:22.847155 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89aa1057-9378-4371-bd8f-70174b2cffee admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:22.849306 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89aa1057-9378-4371-bd8f-70174b2cffee admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:22.867981 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 831/1684] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:25:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:25:22.872911 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48bc5a56-260b-4c19-b41d-7759f2e0c1a1 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:22.886911 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48bc5a56-260b-4c19-b41d-7759f2e0c1a1 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:22.895554 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb47c6b-16ef-4eaf-8d1b-e7a2e3c12ca2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:22.896586 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb47c6b-16ef-4eaf-8d1b-e7a2e3c12ca2 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:22.897092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb47c6b-16ef-4eaf-8d1b-e7a2e3c12ca2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:22.897568 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb47c6b-16ef-4eaf-8d1b-e7a2e3c12ca2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:22.898510 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb47c6b-16ef-4eaf-8d1b-e7a2e3c12ca2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:22.902619 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb47c6b-16ef-4eaf-8d1b-e7a2e3c12ca2 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:22.957003 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 854/1685] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:25:22 2022] POST /identity/v3/projects => generated 375 bytes in 85 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:25:22.961377 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89aa1057-9378-4371-bd8f-70174b2cffee admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:22.974550 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89aa1057-9378-4371-bd8f-70174b2cffee admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:23.120554 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6184b26-5095-4aae-9107-0cca04f6c02d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:23.122127 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6184b26-5095-4aae-9107-0cca04f6c02d admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:23.123411 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6184b26-5095-4aae-9107-0cca04f6c02d admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:23.123887 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6184b26-5095-4aae-9107-0cca04f6c02d admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:23.125692 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6184b26-5095-4aae-9107-0cca04f6c02d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:23.132811 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6184b26-5095-4aae-9107-0cca04f6c02d admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:23.168474 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 832/1686] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:25:22 2022] POST /identity/v3/users => generated 288 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:25:23.172797 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb47c6b-16ef-4eaf-8d1b-e7a2e3c12ca2 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:23.185739 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb47c6b-16ef-4eaf-8d1b-e7a2e3c12ca2 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:23.194419 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd68bf4-11c5-48b6-a642-261bbaec47c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:23.194419 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd68bf4-11c5-48b6-a642-261bbaec47c0 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:23.194419 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd68bf4-11c5-48b6-a642-261bbaec47c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:23.194419 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd68bf4-11c5-48b6-a642-261bbaec47c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:23.194893 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd68bf4-11c5-48b6-a642-261bbaec47c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:23.196482 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd68bf4-11c5-48b6-a642-261bbaec47c0 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:23.211148 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 855/1687] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:25:23 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:25:23.215410 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6184b26-5095-4aae-9107-0cca04f6c02d admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:23.227012 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6184b26-5095-4aae-9107-0cca04f6c02d admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:23.234206 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caca6793-303b-4a24-93f7-0a880321622a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:23.234834 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caca6793-303b-4a24-93f7-0a880321622a admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:23.235203 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caca6793-303b-4a24-93f7-0a880321622a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:23.235368 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caca6793-303b-4a24-93f7-0a880321622a admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:23.236001 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caca6793-303b-4a24-93f7-0a880321622a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:23.237617 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caca6793-303b-4a24-93f7-0a880321622a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:23.252881 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 833/1688] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:25:23 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:25:23.254042 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd68bf4-11c5-48b6-a642-261bbaec47c0 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:23.269593 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd68bf4-11c5-48b6-a642-261bbaec47c0 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:23.272395 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caca6793-303b-4a24-93f7-0a880321622a admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:23.276606 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-caca6793-303b-4a24-93f7-0a880321622a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:23.277787 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d5ccb01-af9a-4a21-9d4f-2435aa5acdc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:23.278829 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5ccb01-af9a-4a21-9d4f-2435aa5acdc9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:23.279318 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5ccb01-af9a-4a21-9d4f-2435aa5acdc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:23.279798 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5ccb01-af9a-4a21-9d4f-2435aa5acdc9 admin admin] PATH_INFO: `/v3/projects/f135389d386e43d58581df01f80c083c/users/2ac8bc3fb2734e979ae50aff1a6e905e/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:23.290638 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caca6793-303b-4a24-93f7-0a880321622a admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:23.295592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e0c84c-41b4-4499-a852-e97bd9e4d668 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:23.297002 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e0c84c-41b4-4499-a852-e97bd9e4d668 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:23.297703 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e0c84c-41b4-4499-a852-e97bd9e4d668 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:23.297986 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e0c84c-41b4-4499-a852-e97bd9e4d668 admin admin] PATH_INFO: `/v3/projects/872936a3af034418b87ab13cfbe191c1` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:23.299170 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e0c84c-41b4-4499-a852-e97bd9e4d668 admin admin] RBAC: Authorizing `identity:delete_project(project_id=872936a3af034418b87ab13cfbe191c1)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:23.301492 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d5ccb01-af9a-4a21-9d4f-2435aa5acdc9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f135389d386e43d58581df01f80c083c, user_id=2ac8bc3fb2734e979ae50aff1a6e905e, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:23.302949 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e0c84c-41b4-4499-a852-e97bd9e4d668 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:23.305039 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d5ccb01-af9a-4a21-9d4f-2435aa5acdc9 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:23.328481 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 856/1689] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:25:23 2022] PUT /identity/v3/projects/f135389d386e43d58581df01f80c083c/users/2ac8bc3fb2734e979ae50aff1a6e905e/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:25:23.340428 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0433f5-9886-43a0-839b-c116bcc13ed2 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:23.340706 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0433f5-9886-43a0-839b-c116bcc13ed2 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:23.340972 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0433f5-9886-43a0-839b-c116bcc13ed2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:23.412060 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-42e0c84c-41b4-4499-a852-e97bd9e4d668 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '872936a3af034418b87ab13cfbe191c1', 'request_id': 'req-42e0c84c-41b4-4499-a852-e97bd9e4d668'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:23.423388 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-9f0433f5-9886-43a0-839b-c116bcc13ed2 None None] MFA Rules not processed for user `2ac8bc3fb2734e979ae50aff1a6e905e`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:23.424763 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-9f0433f5-9886-43a0-839b-c116bcc13ed2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:23.436154 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-42e0c84c-41b4-4499-a852-e97bd9e4d668 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '872936a3af034418b87ab13cfbe191c1', 'request_id': 'req-42e0c84c-41b4-4499-a852-e97bd9e4d668'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:23.459259 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-42e0c84c-41b4-4499-a852-e97bd9e4d668 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '872936a3af034418b87ab13cfbe191c1', 'request_id': 'req-42e0c84c-41b4-4499-a852-e97bd9e4d668'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:23.477517 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-42e0c84c-41b4-4499-a852-e97bd9e4d668 admin admin] The token cache is being invalidate because project 872936a3af034418b87ab13cfbe191c1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:25:23.478405 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-42e0c84c-41b4-4499-a852-e97bd9e4d668 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:23.478764 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-42e0c84c-41b4-4499-a852-e97bd9e4d668 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:23.481550 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 834/1690] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:25:23 2022] DELETE /identity/v3/projects/872936a3af034418b87ab13cfbe191c1 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:23.486330 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e0c84c-41b4-4499-a852-e97bd9e4d668 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:23.487945 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-42e0c84c-41b4-4499-a852-e97bd9e4d668 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:23.494485 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:23.543595 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:23.620730 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 857/1691] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:23 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 281 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:25:23.702792 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f0433f5-9886-43a0-839b-c116bcc13ed2 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:23.704341 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-9f0433f5-9886-43a0-839b-c116bcc13ed2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:23.717093 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e0c84c-41b4-4499-a852-e97bd9e4d668 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:23.728024 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e40d41-2fb7-4236-b23f-a74cbf691e5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:23.728353 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e40d41-2fb7-4236-b23f-a74cbf691e5a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:23.728639 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e40d41-2fb7-4236-b23f-a74cbf691e5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:23.728910 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e40d41-2fb7-4236-b23f-a74cbf691e5a admin admin] PATH_INFO: `/v3/users/189b9109149d4a01a9a3dda68908e03d` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:23.730303 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e40d41-2fb7-4236-b23f-a74cbf691e5a admin admin] RBAC: Authorizing `identity:delete_user(user_id=189b9109149d4a01a9a3dda68908e03d)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:23.734824 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e40d41-2fb7-4236-b23f-a74cbf691e5a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:23.853663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-c6e40d41-2fb7-4236-b23f-a74cbf691e5a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '189b9109149d4a01a9a3dda68908e03d'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:23.870038 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f0433f5-9886-43a0-839b-c116bcc13ed2 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:23.880693 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eac2a17-ad3a-4b48-ada7-cbf02bfc8b1c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:23.883104 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eac2a17-ad3a-4b48-ada7-cbf02bfc8b1c service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:23.883677 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-c6e40d41-2fb7-4236-b23f-a74cbf691e5a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '189b9109149d4a01a9a3dda68908e03d'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:23.884268 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-c6e40d41-2fb7-4236-b23f-a74cbf691e5a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '189b9109149d4a01a9a3dda68908e03d'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:23.884441 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eac2a17-ad3a-4b48-ada7-cbf02bfc8b1c service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:23.884705 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eac2a17-ad3a-4b48-ada7-cbf02bfc8b1c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:23.889315 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-7eac2a17-ad3a-4b48-ada7-cbf02bfc8b1c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:23.916840 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eac2a17-ad3a-4b48-ada7-cbf02bfc8b1c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:23.917210 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-c6e40d41-2fb7-4236-b23f-a74cbf691e5a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '189b9109149d4a01a9a3dda68908e03d'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:23.921975 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eac2a17-ad3a-4b48-ada7-cbf02bfc8b1c service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:23.933796 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-c6e40d41-2fb7-4236-b23f-a74cbf691e5a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '189b9109149d4a01a9a3dda68908e03d'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:23.952950 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 835/1692] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:25:23 2022] DELETE /identity/v3/users/189b9109149d4a01a9a3dda68908e03d => generated 0 bytes in 464 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:23.964647 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 858/1693] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:25:23 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 262 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:25:24.294025 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e40d41-2fb7-4236-b23f-a74cbf691e5a admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:24.296223 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-c6e40d41-2fb7-4236-b23f-a74cbf691e5a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:24.388385 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e40d41-2fb7-4236-b23f-a74cbf691e5a admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:24.395126 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5510a41b-1983-4ba7-8316-d440debc5bcf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:24.398210 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5510a41b-1983-4ba7-8316-d440debc5bcf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:24.398514 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5510a41b-1983-4ba7-8316-d440debc5bcf admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:24.398763 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5510a41b-1983-4ba7-8316-d440debc5bcf admin admin] PATH_INFO: `/v3/projects/652cb71a72974636bd7ef43ad556fcb6` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:24.400034 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5510a41b-1983-4ba7-8316-d440debc5bcf admin admin] RBAC: Authorizing `identity:delete_project(project_id=652cb71a72974636bd7ef43ad556fcb6)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:24.402682 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5510a41b-1983-4ba7-8316-d440debc5bcf admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:24.456942 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:24.509924 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-5510a41b-1983-4ba7-8316-d440debc5bcf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '652cb71a72974636bd7ef43ad556fcb6', 'request_id': 'req-5510a41b-1983-4ba7-8316-d440debc5bcf'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:24.526286 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-5510a41b-1983-4ba7-8316-d440debc5bcf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '652cb71a72974636bd7ef43ad556fcb6', 'request_id': 'req-5510a41b-1983-4ba7-8316-d440debc5bcf'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:24.533721 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-5510a41b-1983-4ba7-8316-d440debc5bcf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '652cb71a72974636bd7ef43ad556fcb6', 'request_id': 'req-5510a41b-1983-4ba7-8316-d440debc5bcf'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:24.554072 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-5510a41b-1983-4ba7-8316-d440debc5bcf admin admin] The token cache is being invalidate because project 652cb71a72974636bd7ef43ad556fcb6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:25:24.554422 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-5510a41b-1983-4ba7-8316-d440debc5bcf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:24.556169 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-5510a41b-1983-4ba7-8316-d440debc5bcf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:24.559236 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 836/1694] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:25:24 2022] DELETE /identity/v3/projects/652cb71a72974636bd7ef43ad556fcb6 => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:24.564161 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6471d76-5fb2-42f2-aeef-dcdaad918692 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:24.564620 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6471d76-5fb2-42f2-aeef-dcdaad918692 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:24.564965 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6471d76-5fb2-42f2-aeef-dcdaad918692 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:24.586105 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-f6471d76-5fb2-42f2-aeef-dcdaad918692 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:24.587365 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-f6471d76-5fb2-42f2-aeef-dcdaad918692 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:24.657107 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 859/1695] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:25:24.662648 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40abd57-db0c-4a80-b7ab-246b39660759 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:24.663148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40abd57-db0c-4a80-b7ab-246b39660759 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:24.663553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40abd57-db0c-4a80-b7ab-246b39660759 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:24.691999 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-f40abd57-db0c-4a80-b7ab-246b39660759 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:24.694014 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f40abd57-db0c-4a80-b7ab-246b39660759 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:24.710397 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 837/1696] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:25:24.711888 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6471d76-5fb2-42f2-aeef-dcdaad918692 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:24.728237 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6471d76-5fb2-42f2-aeef-dcdaad918692 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:24.733554 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b2e000d-83a1-4b48-b737-07ea162d8774 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:24.734180 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2e000d-83a1-4b48-b737-07ea162d8774 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:24.734462 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2e000d-83a1-4b48-b737-07ea162d8774 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:24.734736 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2e000d-83a1-4b48-b737-07ea162d8774 admin admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:24.735397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b2e000d-83a1-4b48-b737-07ea162d8774 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:25:24.735722 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b2e000d-83a1-4b48-b737-07ea162d8774 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:24.737655 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b2e000d-83a1-4b48-b737-07ea162d8774 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:24.754356 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 860/1697] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:25:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:25:24.756696 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40abd57-db0c-4a80-b7ab-246b39660759 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:24.766706 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40abd57-db0c-4a80-b7ab-246b39660759 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:24.775192 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f50558-3dec-467d-a551-5e5db19bab83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:24.776175 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f50558-3dec-467d-a551-5e5db19bab83 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:24.776175 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f50558-3dec-467d-a551-5e5db19bab83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:24.776482 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f50558-3dec-467d-a551-5e5db19bab83 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:24.789323 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f50558-3dec-467d-a551-5e5db19bab83 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:24.793223 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f50558-3dec-467d-a551-5e5db19bab83 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:24.830834 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 838/1698] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:25:24 2022] POST /identity/v3/projects => generated 365 bytes in 75 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:25:24.835069 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b2e000d-83a1-4b48-b737-07ea162d8774 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:24.847628 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b2e000d-83a1-4b48-b737-07ea162d8774 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:24.926009 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9db9ab-2a9f-41c7-ad3a-8576728e0c97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:24.927879 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9db9ab-2a9f-41c7-ad3a-8576728e0c97 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:24.927879 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9db9ab-2a9f-41c7-ad3a-8576728e0c97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:24.928175 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9db9ab-2a9f-41c7-ad3a-8576728e0c97 admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:24.928800 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a9db9ab-2a9f-41c7-ad3a-8576728e0c97 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:24.932323 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a9db9ab-2a9f-41c7-ad3a-8576728e0c97 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:24.943365 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:24.957682 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 861/1699] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:25:24 2022] POST /identity/v3/users => generated 283 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:25:24.961915 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f50558-3dec-467d-a551-5e5db19bab83 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:24.977389 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f50558-3dec-467d-a551-5e5db19bab83 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:24.982652 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc96bc6a-3d77-413d-827a-0f9e3b93c744 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:24.983279 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc96bc6a-3d77-413d-827a-0f9e3b93c744 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:24.983557 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc96bc6a-3d77-413d-827a-0f9e3b93c744 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:24.983830 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc96bc6a-3d77-413d-827a-0f9e3b93c744 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:24.984488 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc96bc6a-3d77-413d-827a-0f9e3b93c744 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:24.986209 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc96bc6a-3d77-413d-827a-0f9e3b93c744 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:24.999486 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 839/1700] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:25:24 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:25:25.003425 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9db9ab-2a9f-41c7-ad3a-8576728e0c97 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:25.016433 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9db9ab-2a9f-41c7-ad3a-8576728e0c97 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:25.024036 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2328bc-b284-4877-bd41-192b5b0626bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:25.024402 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2328bc-b284-4877-bd41-192b5b0626bb admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:25.025483 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2328bc-b284-4877-bd41-192b5b0626bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:25.026863 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2328bc-b284-4877-bd41-192b5b0626bb admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:25.027905 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd2328bc-b284-4877-bd41-192b5b0626bb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:25.030836 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd2328bc-b284-4877-bd41-192b5b0626bb admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:25.047619 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 862/1701] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:25:25 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:25:25.049356 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc96bc6a-3d77-413d-827a-0f9e3b93c744 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:25.061443 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc96bc6a-3d77-413d-827a-0f9e3b93c744 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:25.066563 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9f13f5-ef43-4ed1-8a07-49d7a968df46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:25.068039 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9f13f5-ef43-4ed1-8a07-49d7a968df46 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:25.068537 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9f13f5-ef43-4ed1-8a07-49d7a968df46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:25.068971 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9f13f5-ef43-4ed1-8a07-49d7a968df46 admin admin] PATH_INFO: `/v3/projects/76df20adadcb406dbcac6a9e486c26bc/users/59134275bdd048c09709f9e6897aa1fb/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:25.090873 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd9f13f5-ef43-4ed1-8a07-49d7a968df46 admin admin] RBAC: Authorizing `identity:create_grant(project_id=76df20adadcb406dbcac6a9e486c26bc, user_id=59134275bdd048c09709f9e6897aa1fb, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:25.094052 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd9f13f5-ef43-4ed1-8a07-49d7a968df46 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:25.111825 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 840/1702] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:25:25 2022] PUT /identity/v3/projects/76df20adadcb406dbcac6a9e486c26bc/users/59134275bdd048c09709f9e6897aa1fb/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:25.123447 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c905f445-dfc8-41bf-9d92-2f74015bb5d2 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:25.123789 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c905f445-dfc8-41bf-9d92-2f74015bb5d2 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:25.124134 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c905f445-dfc8-41bf-9d92-2f74015bb5d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:25.159026 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-c905f445-dfc8-41bf-9d92-2f74015bb5d2 None None] MFA Rules not processed for user `59134275bdd048c09709f9e6897aa1fb`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:25.160416 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-c905f445-dfc8-41bf-9d92-2f74015bb5d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:25.243196 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 863/1703] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:25 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:25:25.247902 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9f13f5-ef43-4ed1-8a07-49d7a968df46 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:25.318579 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9f13f5-ef43-4ed1-8a07-49d7a968df46 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:25.323796 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc37138-8307-41ff-a138-3181b4609d99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:25.324415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc37138-8307-41ff-a138-3181b4609d99 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:25.324697 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc37138-8307-41ff-a138-3181b4609d99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:25.324972 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc37138-8307-41ff-a138-3181b4609d99 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:25.325973 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc37138-8307-41ff-a138-3181b4609d99 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:25.327903 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc37138-8307-41ff-a138-3181b4609d99 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:25.386396 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 841/1704] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:25:25 2022] POST /identity/v3/projects => generated 367 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:25:25.387789 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c905f445-dfc8-41bf-9d92-2f74015bb5d2 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:25.406506 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c905f445-dfc8-41bf-9d92-2f74015bb5d2 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:25.539309 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d809358d-07a6-4c44-a866-ff5c8f1ed7c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:25.540032 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d809358d-07a6-4c44-a866-ff5c8f1ed7c2 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:25.540323 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d809358d-07a6-4c44-a866-ff5c8f1ed7c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:25.540587 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d809358d-07a6-4c44-a866-ff5c8f1ed7c2 admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:25.541404 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d809358d-07a6-4c44-a866-ff5c8f1ed7c2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:25.543852 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d809358d-07a6-4c44-a866-ff5c8f1ed7c2 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:25.574894 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 864/1705] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:25:25 2022] POST /identity/v3/users => generated 290 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:25:25.582970 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc37138-8307-41ff-a138-3181b4609d99 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:25.597338 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc37138-8307-41ff-a138-3181b4609d99 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:25.609616 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50af03e4-e54b-4b84-a762-5dcb941f7811 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:25.610299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50af03e4-e54b-4b84-a762-5dcb941f7811 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:25.610586 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50af03e4-e54b-4b84-a762-5dcb941f7811 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:25.611019 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50af03e4-e54b-4b84-a762-5dcb941f7811 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:25.621637 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50af03e4-e54b-4b84-a762-5dcb941f7811 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:25.626626 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50af03e4-e54b-4b84-a762-5dcb941f7811 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:25.641494 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2cd727-76b0-4688-bc98-0c348d91712c None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:25.641688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2cd727-76b0-4688-bc98-0c348d91712c None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:25.642041 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2cd727-76b0-4688-bc98-0c348d91712c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:25.656178 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 842/1706] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:25:25 2022] GET /identity/v3/roles => generated 1420 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:25:25.666492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50af03e4-e54b-4b84-a762-5dcb941f7811 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:25.678312 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-7c2cd727-76b0-4688-bc98-0c348d91712c None None] MFA Rules not processed for user `2ac8bc3fb2734e979ae50aff1a6e905e`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:25.680558 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-7c2cd727-76b0-4688-bc98-0c348d91712c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:25.687301 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50af03e4-e54b-4b84-a762-5dcb941f7811 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:25.709908 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea6af8b2-cd8c-4a91-a99e-8ae34d576b86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:25.710756 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6af8b2-cd8c-4a91-a99e-8ae34d576b86 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:25.711398 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6af8b2-cd8c-4a91-a99e-8ae34d576b86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:25.711685 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6af8b2-cd8c-4a91-a99e-8ae34d576b86 admin admin] PATH_INFO: `/v3/projects/e8f0ca3a9ed74a2d847924798507e1cc/users/042e3ba337ab402b877e77a445566aeb/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:25.740098 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea6af8b2-cd8c-4a91-a99e-8ae34d576b86 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e8f0ca3a9ed74a2d847924798507e1cc, user_id=042e3ba337ab402b877e77a445566aeb, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:25.746094 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea6af8b2-cd8c-4a91-a99e-8ae34d576b86 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:25.787120 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:DELETE=6,keystone:SELECT=89,keystone:INSERT=5 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:25.801275 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 843/1707] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:25:25 2022] PUT /identity/v3/projects/e8f0ca3a9ed74a2d847924798507e1cc/users/042e3ba337ab402b877e77a445566aeb/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:25.801275 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea6af8b2-cd8c-4a91-a99e-8ae34d576b86 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:25.849164 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 865/1708] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:25 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:25:25.894936 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7670f3fa-f178-4df0-943a-61ad99993418 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:25.895375 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7670f3fa-f178-4df0-943a-61ad99993418 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:25.896799 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7670f3fa-f178-4df0-943a-61ad99993418 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:25.898008 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 866/1709] 10.222.0.42 () {54 vars in 921 bytes} [Thu Oct 27 01:25:25 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:25:25.930598 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea6af8b2-cd8c-4a91-a99e-8ae34d576b86 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:25.941635 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67574f5e-87b4-4896-8606-44bd0a18a5f8 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:25.942149 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67574f5e-87b4-4896-8606-44bd0a18a5f8 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:25.942862 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67574f5e-87b4-4896-8606-44bd0a18a5f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:25.943373 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307aefa6-6076-42c7-a56c-ecea9b66764b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:25.944226 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307aefa6-6076-42c7-a56c-ecea9b66764b admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:25.944512 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307aefa6-6076-42c7-a56c-ecea9b66764b admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:25.945476 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307aefa6-6076-42c7-a56c-ecea9b66764b admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:25.946487 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-307aefa6-6076-42c7-a56c-ecea9b66764b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:25.949214 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-307aefa6-6076-42c7-a56c-ecea9b66764b admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:25.962149 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:INSERT=5,keystone:SELECT=95 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:25.966926 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 844/1710] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:25:25 2022] GET /identity/v3/roles => generated 1420 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:25:25.971905 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307aefa6-6076-42c7-a56c-ecea9b66764b admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:25.989494 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307aefa6-6076-42c7-a56c-ecea9b66764b admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:25.990967 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-67574f5e-87b4-4896-8606-44bd0a18a5f8 None None] MFA Rules not processed for user `62051562fc0b4cd08750d2293e71787c`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:25.992659 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-67574f5e-87b4-4896-8606-44bd0a18a5f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:25.996039 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67df2b31-2829-4876-938b-d9fe5c7e325c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:25.996994 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67df2b31-2829-4876-938b-d9fe5c7e325c admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:25.997327 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67df2b31-2829-4876-938b-d9fe5c7e325c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:25.997600 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67df2b31-2829-4876-938b-d9fe5c7e325c admin admin] PATH_INFO: `/v3/projects/e8f0ca3a9ed74a2d847924798507e1cc/users/042e3ba337ab402b877e77a445566aeb/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:25.999740 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67df2b31-2829-4876-938b-d9fe5c7e325c admin admin] RBAC: Authorizing `identity:create_grant(project_id=e8f0ca3a9ed74a2d847924798507e1cc, user_id=042e3ba337ab402b877e77a445566aeb, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:26.003712 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67df2b31-2829-4876-938b-d9fe5c7e325c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:26.026520 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 845/1711] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:25:25 2022] PUT /identity/v3/projects/e8f0ca3a9ed74a2d847924798507e1cc/users/042e3ba337ab402b877e77a445566aeb/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:26.031153 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf27cc9-6d4f-483d-98e7-96c6abea3ec2 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:26.031503 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf27cc9-6d4f-483d-98e7-96c6abea3ec2 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:26.031854 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf27cc9-6d4f-483d-98e7-96c6abea3ec2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:26.089731 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-5cf27cc9-6d4f-483d-98e7-96c6abea3ec2 None None] MFA Rules not processed for user `042e3ba337ab402b877e77a445566aeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:26.100005 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-5cf27cc9-6d4f-483d-98e7-96c6abea3ec2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:26.115459 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 867/1712] 10.222.0.42 () {58 vars in 1018 bytes} [Thu Oct 27 01:25:25 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:25:26.133946 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67574f5e-87b4-4896-8606-44bd0a18a5f8 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:26.135485 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-67574f5e-87b4-4896-8606-44bd0a18a5f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:26.159200 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67574f5e-87b4-4896-8606-44bd0a18a5f8 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:26.215632 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 846/1713] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:26 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:25:26.228436 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf27cc9-6d4f-483d-98e7-96c6abea3ec2 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:26.229917 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-5cf27cc9-6d4f-483d-98e7-96c6abea3ec2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:26.290186 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b96b105-1c0c-4c27-a305-2bbee6da2151 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62051562fc0b4cd08750d2293e71787c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:26.290820 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b96b105-1c0c-4c27-a305-2bbee6da2151 service placement] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:26.291103 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b96b105-1c0c-4c27-a305-2bbee6da2151 service placement] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:26.291375 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b96b105-1c0c-4c27-a305-2bbee6da2151 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:26.303101 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b96b105-1c0c-4c27-a305-2bbee6da2151 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:26.304709 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b96b105-1c0c-4c27-a305-2bbee6da2151 service placement] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:26.325922 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 868/1714] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:25:26 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 192 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 27 01:25:26.364531 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf27cc9-6d4f-483d-98e7-96c6abea3ec2 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:26.379054 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a38b0e5-9409-4033-b06d-190ed1bed58f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:26.380480 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a38b0e5-9409-4033-b06d-190ed1bed58f service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:26.380997 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a38b0e5-9409-4033-b06d-190ed1bed58f service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:26.381535 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a38b0e5-9409-4033-b06d-190ed1bed58f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:26.402136 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a38b0e5-9409-4033-b06d-190ed1bed58f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:26.404446 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a38b0e5-9409-4033-b06d-190ed1bed58f service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:26.415010 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 847/1715] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:25:26 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 187 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:25:26.478028 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b96b105-1c0c-4c27-a305-2bbee6da2151 service placement] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:26.480961 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-9b96b105-1c0c-4c27-a305-2bbee6da2151 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:26.494833 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b96b105-1c0c-4c27-a305-2bbee6da2151 service placement] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:26.499413 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f50b10c-cff8-4cfb-a284-429788790422 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:26.500458 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f50b10c-cff8-4cfb-a284-429788790422 service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:26.500869 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f50b10c-cff8-4cfb-a284-429788790422 service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:26.501480 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f50b10c-cff8-4cfb-a284-429788790422 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:26.578785 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f50b10c-cff8-4cfb-a284-429788790422 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:26.580403 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f50b10c-cff8-4cfb-a284-429788790422 service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:26.594182 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 869/1716] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:25:26 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:25:27.287957 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a38b0e5-9409-4033-b06d-190ed1bed58f service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:27.290271 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0a38b0e5-9409-4033-b06d-190ed1bed58f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:27.309328 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:27.374612 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a38b0e5-9409-4033-b06d-190ed1bed58f service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:27.380127 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ff173b-b303-43bc-a880-5603afb16c67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:27.380769 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ff173b-b303-43bc-a880-5603afb16c67 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:27.381161 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ff173b-b303-43bc-a880-5603afb16c67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:27.381434 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ff173b-b303-43bc-a880-5603afb16c67 admin admin] PATH_INFO: `/v3/users/248eb53661d24b3380de06b7f6b10dd8` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:27.382837 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83ff173b-b303-43bc-a880-5603afb16c67 admin admin] RBAC: Authorizing `identity:delete_user(user_id=248eb53661d24b3380de06b7f6b10dd8)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:27.384477 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83ff173b-b303-43bc-a880-5603afb16c67 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:27.450340 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-83ff173b-b303-43bc-a880-5603afb16c67 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '248eb53661d24b3380de06b7f6b10dd8'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:27.458289 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-83ff173b-b303-43bc-a880-5603afb16c67 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '248eb53661d24b3380de06b7f6b10dd8'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:27.458947 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-83ff173b-b303-43bc-a880-5603afb16c67 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '248eb53661d24b3380de06b7f6b10dd8'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:27.471574 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-83ff173b-b303-43bc-a880-5603afb16c67 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '248eb53661d24b3380de06b7f6b10dd8'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:27.482102 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-83ff173b-b303-43bc-a880-5603afb16c67 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '248eb53661d24b3380de06b7f6b10dd8'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:27.488584 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 848/1717] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:25:27 2022] DELETE /identity/v3/users/248eb53661d24b3380de06b7f6b10dd8 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:25:27.664526 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f50b10c-cff8-4cfb-a284-429788790422 service cinder] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:27.677751 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f50b10c-cff8-4cfb-a284-429788790422 service cinder] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:27.710277 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:27.775632 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250697fa-7ae7-4cb2-ab14-ed124b44d428 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:27.776373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250697fa-7ae7-4cb2-ab14-ed124b44d428 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:27.776655 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250697fa-7ae7-4cb2-ab14-ed124b44d428 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:27.777129 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250697fa-7ae7-4cb2-ab14-ed124b44d428 admin admin] PATH_INFO: `/v3/projects/c3e3b33974604b8f8cdb5f4849917a4b` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:27.778352 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-250697fa-7ae7-4cb2-ab14-ed124b44d428 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c3e3b33974604b8f8cdb5f4849917a4b)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:27.781032 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-250697fa-7ae7-4cb2-ab14-ed124b44d428 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:27.864192 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-250697fa-7ae7-4cb2-ab14-ed124b44d428 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c3e3b33974604b8f8cdb5f4849917a4b', 'request_id': 'req-250697fa-7ae7-4cb2-ab14-ed124b44d428'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:27.874773 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-250697fa-7ae7-4cb2-ab14-ed124b44d428 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c3e3b33974604b8f8cdb5f4849917a4b', 'request_id': 'req-250697fa-7ae7-4cb2-ab14-ed124b44d428'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:27.886879 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-250697fa-7ae7-4cb2-ab14-ed124b44d428 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c3e3b33974604b8f8cdb5f4849917a4b', 'request_id': 'req-250697fa-7ae7-4cb2-ab14-ed124b44d428'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:27.914135 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-250697fa-7ae7-4cb2-ab14-ed124b44d428 admin admin] The token cache is being invalidate because project c3e3b33974604b8f8cdb5f4849917a4b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:25:27.914504 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-250697fa-7ae7-4cb2-ab14-ed124b44d428 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:27.915929 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-250697fa-7ae7-4cb2-ab14-ed124b44d428 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:27.920027 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 870/1718] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:25:27 2022] DELETE /identity/v3/projects/c3e3b33974604b8f8cdb5f4849917a4b => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:27.925262 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ff173b-b303-43bc-a880-5603afb16c67 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:27.927242 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-83ff173b-b303-43bc-a880-5603afb16c67 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:27.940462 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ff173b-b303-43bc-a880-5603afb16c67 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:28.034619 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4afacfd4-c6e2-43ec-a600-d975f88dae48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:28.035465 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4afacfd4-c6e2-43ec-a600-d975f88dae48 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:28.036356 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4afacfd4-c6e2-43ec-a600-d975f88dae48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:28.036931 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4afacfd4-c6e2-43ec-a600-d975f88dae48 admin admin] PATH_INFO: `/v3/users/6bd65bfc7c994b58a47b2314bedd8c84` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:28.038382 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4afacfd4-c6e2-43ec-a600-d975f88dae48 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6bd65bfc7c994b58a47b2314bedd8c84)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:28.040427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4afacfd4-c6e2-43ec-a600-d975f88dae48 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:28.118504 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:28.139648 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4afacfd4-c6e2-43ec-a600-d975f88dae48 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6bd65bfc7c994b58a47b2314bedd8c84'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:28.148984 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4afacfd4-c6e2-43ec-a600-d975f88dae48 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6bd65bfc7c994b58a47b2314bedd8c84'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:28.149641 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4afacfd4-c6e2-43ec-a600-d975f88dae48 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6bd65bfc7c994b58a47b2314bedd8c84'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:28.162465 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4afacfd4-c6e2-43ec-a600-d975f88dae48 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6bd65bfc7c994b58a47b2314bedd8c84'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:28.169509 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4afacfd4-c6e2-43ec-a600-d975f88dae48 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6bd65bfc7c994b58a47b2314bedd8c84'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:28.176057 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 849/1719] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:25:27 2022] DELETE /identity/v3/users/6bd65bfc7c994b58a47b2314bedd8c84 => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:28.501701 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250697fa-7ae7-4cb2-ab14-ed124b44d428 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:28.503215 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-250697fa-7ae7-4cb2-ab14-ed124b44d428 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:28.516823 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250697fa-7ae7-4cb2-ab14-ed124b44d428 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:28.599941 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ff7079-7985-4c9e-b3ea-95ce50a84fc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:28.599941 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ff7079-7985-4c9e-b3ea-95ce50a84fc6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:28.599941 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ff7079-7985-4c9e-b3ea-95ce50a84fc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:28.599941 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ff7079-7985-4c9e-b3ea-95ce50a84fc6 admin admin] PATH_INFO: `/v3/projects/f6f36b70f1774b18b5472730833fd38c` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:28.600652 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ff7079-7985-4c9e-b3ea-95ce50a84fc6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f6f36b70f1774b18b5472730833fd38c)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:28.602667 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ff7079-7985-4c9e-b3ea-95ce50a84fc6 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:28.663141 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-42ff7079-7985-4c9e-b3ea-95ce50a84fc6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f6f36b70f1774b18b5472730833fd38c', 'request_id': 'req-42ff7079-7985-4c9e-b3ea-95ce50a84fc6'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:28.676548 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-42ff7079-7985-4c9e-b3ea-95ce50a84fc6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f6f36b70f1774b18b5472730833fd38c', 'request_id': 'req-42ff7079-7985-4c9e-b3ea-95ce50a84fc6'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:28.685681 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-42ff7079-7985-4c9e-b3ea-95ce50a84fc6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f6f36b70f1774b18b5472730833fd38c', 'request_id': 'req-42ff7079-7985-4c9e-b3ea-95ce50a84fc6'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:28.687520 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:28.706287 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-42ff7079-7985-4c9e-b3ea-95ce50a84fc6 admin admin] The token cache is being invalidate because project f6f36b70f1774b18b5472730833fd38c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:25:28.706729 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-42ff7079-7985-4c9e-b3ea-95ce50a84fc6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:28.707534 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-42ff7079-7985-4c9e-b3ea-95ce50a84fc6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:28.710618 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 871/1720] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:25:28 2022] DELETE /identity/v3/projects/f6f36b70f1774b18b5472730833fd38c => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:28.715145 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db6abba-fb6d-43d1-aa81-6a521f72cfbf None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:28.715527 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db6abba-fb6d-43d1-aa81-6a521f72cfbf None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:28.715698 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db6abba-fb6d-43d1-aa81-6a521f72cfbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:28.759913 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-4db6abba-fb6d-43d1-aa81-6a521f72cfbf None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:28.761259 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-4db6abba-fb6d-43d1-aa81-6a521f72cfbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:28.863296 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 850/1721] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:25:28.867331 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08898f3b-724b-4c9d-b7b3-751d43225388 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:28.867619 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08898f3b-724b-4c9d-b7b3-751d43225388 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:28.867897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08898f3b-724b-4c9d-b7b3-751d43225388 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:28.889279 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-08898f3b-724b-4c9d-b7b3-751d43225388 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:28.890444 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-08898f3b-724b-4c9d-b7b3-751d43225388 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:28.898839 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 872/1722] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:25:28.903036 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db6abba-fb6d-43d1-aa81-6a521f72cfbf None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:28.914315 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db6abba-fb6d-43d1-aa81-6a521f72cfbf None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:28.918990 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf9da30-2bbc-4a13-8fb0-43aeccb2b336 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:28.919598 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf9da30-2bbc-4a13-8fb0-43aeccb2b336 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:28.919877 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf9da30-2bbc-4a13-8fb0-43aeccb2b336 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:28.920151 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf9da30-2bbc-4a13-8fb0-43aeccb2b336 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:28.920829 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf9da30-2bbc-4a13-8fb0-43aeccb2b336 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:25:28.921168 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf9da30-2bbc-4a13-8fb0-43aeccb2b336 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:28.922769 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf9da30-2bbc-4a13-8fb0-43aeccb2b336 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:28.939495 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 851/1723] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:25:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:25:28.940682 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08898f3b-724b-4c9d-b7b3-751d43225388 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:28.951018 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08898f3b-724b-4c9d-b7b3-751d43225388 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:28.956508 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebf5fa2-9af3-4022-a721-ce991c222877 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:28.957509 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebf5fa2-9af3-4022-a721-ce991c222877 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:28.957509 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebf5fa2-9af3-4022-a721-ce991c222877 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:28.958029 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebf5fa2-9af3-4022-a721-ce991c222877 admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:28.958830 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ebf5fa2-9af3-4022-a721-ce991c222877 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:28.960751 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ebf5fa2-9af3-4022-a721-ce991c222877 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:29.000844 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 873/1724] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:25:28 2022] POST /identity/v3/projects => generated 375 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:25:29.006667 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf9da30-2bbc-4a13-8fb0-43aeccb2b336 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:29.017003 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf9da30-2bbc-4a13-8fb0-43aeccb2b336 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:29.120758 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e63eaa3-fba1-4b4b-970e-bfcc925e9da2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:29.121603 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e63eaa3-fba1-4b4b-970e-bfcc925e9da2 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:29.121945 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e63eaa3-fba1-4b4b-970e-bfcc925e9da2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:29.122303 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e63eaa3-fba1-4b4b-970e-bfcc925e9da2 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:29.123026 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e63eaa3-fba1-4b4b-970e-bfcc925e9da2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:29.125042 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e63eaa3-fba1-4b4b-970e-bfcc925e9da2 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:29.185848 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 852/1725] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:25:29 2022] POST /identity/v3/users => generated 288 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:25:29.202245 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebf5fa2-9af3-4022-a721-ce991c222877 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:29.223074 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebf5fa2-9af3-4022-a721-ce991c222877 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:29.231518 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3fcd810-b13a-4ade-993f-f6e361a2983b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:29.231847 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fcd810-b13a-4ade-993f-f6e361a2983b admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:29.232096 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fcd810-b13a-4ade-993f-f6e361a2983b admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:29.232456 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fcd810-b13a-4ade-993f-f6e361a2983b admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:29.233374 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3fcd810-b13a-4ade-993f-f6e361a2983b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:29.235253 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3fcd810-b13a-4ade-993f-f6e361a2983b admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:29.249111 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 874/1726] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:25:29 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:25:29.253528 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e63eaa3-fba1-4b4b-970e-bfcc925e9da2 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:29.267740 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e63eaa3-fba1-4b4b-970e-bfcc925e9da2 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:29.275222 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c262458a-8a9a-49ce-8771-b05f73edb133 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:29.275839 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c262458a-8a9a-49ce-8771-b05f73edb133 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:29.276120 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c262458a-8a9a-49ce-8771-b05f73edb133 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:29.276392 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c262458a-8a9a-49ce-8771-b05f73edb133 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:29.277086 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c262458a-8a9a-49ce-8771-b05f73edb133 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:29.278725 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c262458a-8a9a-49ce-8771-b05f73edb133 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:29.294353 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 853/1727] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:25:29 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:25:29.298512 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3fcd810-b13a-4ade-993f-f6e361a2983b admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:29.309912 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3fcd810-b13a-4ade-993f-f6e361a2983b admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:29.316719 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d4230d-7a1c-44b4-86d9-9e5b1544dd91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:29.316719 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d4230d-7a1c-44b4-86d9-9e5b1544dd91 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:29.316719 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d4230d-7a1c-44b4-86d9-9e5b1544dd91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:29.316719 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d4230d-7a1c-44b4-86d9-9e5b1544dd91 admin admin] PATH_INFO: `/v3/projects/96f48dca6fc54975affc87292e52e111/users/8d73612498fa4689aae23096ab5dfa1c/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:29.346478 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d4230d-7a1c-44b4-86d9-9e5b1544dd91 admin admin] RBAC: Authorizing `identity:create_grant(project_id=96f48dca6fc54975affc87292e52e111, user_id=8d73612498fa4689aae23096ab5dfa1c, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:29.351500 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d4230d-7a1c-44b4-86d9-9e5b1544dd91 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:29.368308 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 875/1728] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:25:29 2022] PUT /identity/v3/projects/96f48dca6fc54975affc87292e52e111/users/8d73612498fa4689aae23096ab5dfa1c/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:29.378643 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0e20da-3e39-4e84-bbdd-57cdc25a4cea None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:29.378907 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0e20da-3e39-4e84-bbdd-57cdc25a4cea None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:29.379382 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0e20da-3e39-4e84-bbdd-57cdc25a4cea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:29.409705 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-4d0e20da-3e39-4e84-bbdd-57cdc25a4cea None None] MFA Rules not processed for user `8d73612498fa4689aae23096ab5dfa1c`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:29.410935 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-4d0e20da-3e39-4e84-bbdd-57cdc25a4cea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:29.416030 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:29.483737 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 854/1729] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:29 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:25:29.488305 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d4230d-7a1c-44b4-86d9-9e5b1544dd91 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:29.561533 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d4230d-7a1c-44b4-86d9-9e5b1544dd91 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:29.566099 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a35fec-a9cd-4541-80b9-8122537a0085 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:29.566701 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a35fec-a9cd-4541-80b9-8122537a0085 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:29.566972 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a35fec-a9cd-4541-80b9-8122537a0085 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:29.567862 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a35fec-a9cd-4541-80b9-8122537a0085 admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:29.569175 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a35fec-a9cd-4541-80b9-8122537a0085 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:29.571073 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a35fec-a9cd-4541-80b9-8122537a0085 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:29.599407 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 876/1730] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:25:29 2022] POST /identity/v3/projects => generated 375 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:25:29.603347 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0e20da-3e39-4e84-bbdd-57cdc25a4cea None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:29.614881 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0e20da-3e39-4e84-bbdd-57cdc25a4cea None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:29.688031 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a197e2-b821-44dd-be3f-cee43ae92ad1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:29.688744 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a197e2-b821-44dd-be3f-cee43ae92ad1 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:29.689029 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a197e2-b821-44dd-be3f-cee43ae92ad1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:29.689335 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a197e2-b821-44dd-be3f-cee43ae92ad1 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:29.690001 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a197e2-b821-44dd-be3f-cee43ae92ad1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:29.691927 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a197e2-b821-44dd-be3f-cee43ae92ad1 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:29.716728 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 855/1731] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:25:29 2022] POST /identity/v3/users => generated 294 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:25:29.720420 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a35fec-a9cd-4541-80b9-8122537a0085 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:29.731263 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a35fec-a9cd-4541-80b9-8122537a0085 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:29.736209 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3df764f-0821-4486-ba8d-255cf15ae86e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:29.737099 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3df764f-0821-4486-ba8d-255cf15ae86e admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:29.737329 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3df764f-0821-4486-ba8d-255cf15ae86e admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:29.737708 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3df764f-0821-4486-ba8d-255cf15ae86e admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:29.738457 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3df764f-0821-4486-ba8d-255cf15ae86e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:29.740677 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3df764f-0821-4486-ba8d-255cf15ae86e admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:29.753105 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 877/1732] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:25:29 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:25:29.759521 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a197e2-b821-44dd-be3f-cee43ae92ad1 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:29.773975 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a197e2-b821-44dd-be3f-cee43ae92ad1 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:29.780844 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae2526d-8dcc-4f66-86cf-fce848dde5e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:29.781659 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae2526d-8dcc-4f66-86cf-fce848dde5e1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:29.782230 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae2526d-8dcc-4f66-86cf-fce848dde5e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:29.782352 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae2526d-8dcc-4f66-86cf-fce848dde5e1 admin admin] PATH_INFO: `/v3/projects/d28bf017913d46819ed954f74d8d1d34/users/fa80d74e22fe4578af8f994818681496/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:29.815193 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae2526d-8dcc-4f66-86cf-fce848dde5e1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d28bf017913d46819ed954f74d8d1d34, user_id=fa80d74e22fe4578af8f994818681496, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:29.818153 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae2526d-8dcc-4f66-86cf-fce848dde5e1 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:29.840297 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 856/1733] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:25:29 2022] PUT /identity/v3/projects/d28bf017913d46819ed954f74d8d1d34/users/fa80d74e22fe4578af8f994818681496/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:25:29.841474 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3df764f-0821-4486-ba8d-255cf15ae86e admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:29.858751 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3df764f-0821-4486-ba8d-255cf15ae86e admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:29.942718 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e15b6a2-993f-4765-848f-e369e95e556d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:29.943570 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e15b6a2-993f-4765-848f-e369e95e556d admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:29.943948 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e15b6a2-993f-4765-848f-e369e95e556d admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:29.944895 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e15b6a2-993f-4765-848f-e369e95e556d admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:29.948898 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e15b6a2-993f-4765-848f-e369e95e556d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:29.948898 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e15b6a2-993f-4765-848f-e369e95e556d admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:29.950193 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:DELETE=2,keystone:SELECT=95,keystone:INSERT=3 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:29.964146 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 878/1734] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:25:29 2022] GET /identity/v3/roles => generated 1420 bytes in 120 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:25:29.965419 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae2526d-8dcc-4f66-86cf-fce848dde5e1 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:29.977969 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae2526d-8dcc-4f66-86cf-fce848dde5e1 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:29.984585 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a64d38-c12d-4f44-934d-bae812ad0f64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:29.985874 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a64d38-c12d-4f44-934d-bae812ad0f64 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:29.986160 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a64d38-c12d-4f44-934d-bae812ad0f64 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:29.986439 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a64d38-c12d-4f44-934d-bae812ad0f64 admin admin] PATH_INFO: `/v3/projects/d28bf017913d46819ed954f74d8d1d34/users/fa80d74e22fe4578af8f994818681496/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:29.989525 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2a64d38-c12d-4f44-934d-bae812ad0f64 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d28bf017913d46819ed954f74d8d1d34, user_id=fa80d74e22fe4578af8f994818681496, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:29.993190 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2a64d38-c12d-4f44-934d-bae812ad0f64 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:30.009880 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 857/1735] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:25:29 2022] PUT /identity/v3/projects/d28bf017913d46819ed954f74d8d1d34/users/fa80d74e22fe4578af8f994818681496/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 45 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:30.022483 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d683916e-3237-402a-923a-594145380cf3 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:30.022911 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d683916e-3237-402a-923a-594145380cf3 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:30.023183 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d683916e-3237-402a-923a-594145380cf3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:30.053869 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-d683916e-3237-402a-923a-594145380cf3 None None] MFA Rules not processed for user `fa80d74e22fe4578af8f994818681496`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:30.056969 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-d683916e-3237-402a-923a-594145380cf3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:30.149770 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 879/1736] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:30 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:25:30.159088 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a64d38-c12d-4f44-934d-bae812ad0f64 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:30.160932 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e2a64d38-c12d-4f44-934d-bae812ad0f64 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:30.228465 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a64d38-c12d-4f44-934d-bae812ad0f64 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:30.232325 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be64d05a-ae4b-49ba-9783-cb9846f3f5ec service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:30.232931 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be64d05a-ae4b-49ba-9783-cb9846f3f5ec service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:30.233246 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be64d05a-ae4b-49ba-9783-cb9846f3f5ec service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:30.233520 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be64d05a-ae4b-49ba-9783-cb9846f3f5ec service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:30.244640 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:30.314305 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be64d05a-ae4b-49ba-9783-cb9846f3f5ec service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:30.315971 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be64d05a-ae4b-49ba-9783-cb9846f3f5ec service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:30.329578 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 858/1737] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:25:30 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 171 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:25:30.526933 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d683916e-3237-402a-923a-594145380cf3 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:30.531748 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-d683916e-3237-402a-923a-594145380cf3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:30.661545 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d683916e-3237-402a-923a-594145380cf3 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:30.666976 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a558c264-572e-47cc-9201-f8ba72ba4582 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:30.667981 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a558c264-572e-47cc-9201-f8ba72ba4582 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:30.668288 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a558c264-572e-47cc-9201-f8ba72ba4582 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:30.668687 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a558c264-572e-47cc-9201-f8ba72ba4582 admin admin] PATH_INFO: `/v3/users/59134275bdd048c09709f9e6897aa1fb` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:30.669943 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a558c264-572e-47cc-9201-f8ba72ba4582 admin admin] RBAC: Authorizing `identity:delete_user(user_id=59134275bdd048c09709f9e6897aa1fb)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:30.671964 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a558c264-572e-47cc-9201-f8ba72ba4582 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:30.730975 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-a558c264-572e-47cc-9201-f8ba72ba4582 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '59134275bdd048c09709f9e6897aa1fb'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:30.741527 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-a558c264-572e-47cc-9201-f8ba72ba4582 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '59134275bdd048c09709f9e6897aa1fb'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:30.759188 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-a558c264-572e-47cc-9201-f8ba72ba4582 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '59134275bdd048c09709f9e6897aa1fb'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:30.760823 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-a558c264-572e-47cc-9201-f8ba72ba4582 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '59134275bdd048c09709f9e6897aa1fb'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:30.761776 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-a558c264-572e-47cc-9201-f8ba72ba4582 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '59134275bdd048c09709f9e6897aa1fb'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:30.775098 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:30.783895 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 880/1738] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:25:30 2022] DELETE /identity/v3/users/59134275bdd048c09709f9e6897aa1fb => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:30.789903 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9843176-b110-4975-be25-d7b5ecb3d156 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:30.790387 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9843176-b110-4975-be25-d7b5ecb3d156 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:30.791931 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9843176-b110-4975-be25-d7b5ecb3d156 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:30.824581 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-f9843176-b110-4975-be25-d7b5ecb3d156 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:30.826299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f9843176-b110-4975-be25-d7b5ecb3d156 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:30.894449 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 859/1739] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:25:30.904889 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a558c264-572e-47cc-9201-f8ba72ba4582 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:30.906407 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-a558c264-572e-47cc-9201-f8ba72ba4582 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:30.975458 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a558c264-572e-47cc-9201-f8ba72ba4582 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:30.979191 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f3e929-b4db-4958-9b4e-234e4612055d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:30.979949 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f3e929-b4db-4958-9b4e-234e4612055d service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:30.980229 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f3e929-b4db-4958-9b4e-234e4612055d service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:30.980499 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f3e929-b4db-4958-9b4e-234e4612055d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:30.992012 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f3e929-b4db-4958-9b4e-234e4612055d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:30.993496 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f3e929-b4db-4958-9b4e-234e4612055d service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:31.005125 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 881/1740] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:25:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:25:31.286781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9843176-b110-4975-be25-d7b5ecb3d156 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:31.288520 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f9843176-b110-4975-be25-d7b5ecb3d156 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:31.318347 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9843176-b110-4975-be25-d7b5ecb3d156 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:31.327162 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd338a13-0e51-4e49-b07b-b06747a2ebcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:31.328164 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd338a13-0e51-4e49-b07b-b06747a2ebcc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:31.328638 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd338a13-0e51-4e49-b07b-b06747a2ebcc admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:31.329116 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd338a13-0e51-4e49-b07b-b06747a2ebcc admin admin] PATH_INFO: `/v3/projects/76df20adadcb406dbcac6a9e486c26bc` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:31.330776 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd338a13-0e51-4e49-b07b-b06747a2ebcc admin admin] RBAC: Authorizing `identity:delete_project(project_id=76df20adadcb406dbcac6a9e486c26bc)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:31.335290 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd338a13-0e51-4e49-b07b-b06747a2ebcc admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:31.396110 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-cd338a13-0e51-4e49-b07b-b06747a2ebcc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '76df20adadcb406dbcac6a9e486c26bc', 'request_id': 'req-cd338a13-0e51-4e49-b07b-b06747a2ebcc'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:31.404075 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-cd338a13-0e51-4e49-b07b-b06747a2ebcc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '76df20adadcb406dbcac6a9e486c26bc', 'request_id': 'req-cd338a13-0e51-4e49-b07b-b06747a2ebcc'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:31.409829 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-cd338a13-0e51-4e49-b07b-b06747a2ebcc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '76df20adadcb406dbcac6a9e486c26bc', 'request_id': 'req-cd338a13-0e51-4e49-b07b-b06747a2ebcc'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:31.422293 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-cd338a13-0e51-4e49-b07b-b06747a2ebcc admin admin] The token cache is being invalidate because project 76df20adadcb406dbcac6a9e486c26bc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:25:31.422605 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-cd338a13-0e51-4e49-b07b-b06747a2ebcc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:31.423249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-cd338a13-0e51-4e49-b07b-b06747a2ebcc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:31.424830 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 860/1741] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:25:31 2022] DELETE /identity/v3/projects/76df20adadcb406dbcac6a9e486c26bc => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:31.429594 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f3e929-b4db-4958-9b4e-234e4612055d service neutron] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:31.440165 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f3e929-b4db-4958-9b4e-234e4612055d service neutron] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:31.500844 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01dfcf97-b60b-4ca3-a21c-ff12f5b5ff41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:31.501540 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01dfcf97-b60b-4ca3-a21c-ff12f5b5ff41 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:31.501822 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01dfcf97-b60b-4ca3-a21c-ff12f5b5ff41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:31.502097 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01dfcf97-b60b-4ca3-a21c-ff12f5b5ff41 admin admin] PATH_INFO: `/v3/users/042e3ba337ab402b877e77a445566aeb` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:31.503302 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01dfcf97-b60b-4ca3-a21c-ff12f5b5ff41 admin admin] RBAC: Authorizing `identity:delete_user(user_id=042e3ba337ab402b877e77a445566aeb)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:31.505218 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01dfcf97-b60b-4ca3-a21c-ff12f5b5ff41 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:31.558346 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-01dfcf97-b60b-4ca3-a21c-ff12f5b5ff41 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '042e3ba337ab402b877e77a445566aeb'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:31.566839 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-01dfcf97-b60b-4ca3-a21c-ff12f5b5ff41 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '042e3ba337ab402b877e77a445566aeb'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:31.583988 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-01dfcf97-b60b-4ca3-a21c-ff12f5b5ff41 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '042e3ba337ab402b877e77a445566aeb'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:31.585524 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-01dfcf97-b60b-4ca3-a21c-ff12f5b5ff41 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '042e3ba337ab402b877e77a445566aeb'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:31.585524 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-01dfcf97-b60b-4ca3-a21c-ff12f5b5ff41 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '042e3ba337ab402b877e77a445566aeb'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:31.604638 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 882/1742] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:25:31 2022] DELETE /identity/v3/users/042e3ba337ab402b877e77a445566aeb => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:32.229977 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd338a13-0e51-4e49-b07b-b06747a2ebcc admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:32.232176 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-cd338a13-0e51-4e49-b07b-b06747a2ebcc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:32.258878 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd338a13-0e51-4e49-b07b-b06747a2ebcc admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:32.306917 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:32.340388 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05dc2006-54cb-4f8d-97a3-6016a13270ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:32.341582 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05dc2006-54cb-4f8d-97a3-6016a13270ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:32.341964 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05dc2006-54cb-4f8d-97a3-6016a13270ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:32.342248 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05dc2006-54cb-4f8d-97a3-6016a13270ab admin admin] PATH_INFO: `/v3/projects/e8f0ca3a9ed74a2d847924798507e1cc` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:32.343572 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05dc2006-54cb-4f8d-97a3-6016a13270ab admin admin] RBAC: Authorizing `identity:delete_project(project_id=e8f0ca3a9ed74a2d847924798507e1cc)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:32.346654 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05dc2006-54cb-4f8d-97a3-6016a13270ab admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:32.417375 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-05dc2006-54cb-4f8d-97a3-6016a13270ab admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e8f0ca3a9ed74a2d847924798507e1cc', 'request_id': 'req-05dc2006-54cb-4f8d-97a3-6016a13270ab'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:32.431782 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-05dc2006-54cb-4f8d-97a3-6016a13270ab admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e8f0ca3a9ed74a2d847924798507e1cc', 'request_id': 'req-05dc2006-54cb-4f8d-97a3-6016a13270ab'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:32.446510 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-05dc2006-54cb-4f8d-97a3-6016a13270ab admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e8f0ca3a9ed74a2d847924798507e1cc', 'request_id': 'req-05dc2006-54cb-4f8d-97a3-6016a13270ab'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:32.461947 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-05dc2006-54cb-4f8d-97a3-6016a13270ab admin admin] The token cache is being invalidate because project e8f0ca3a9ed74a2d847924798507e1cc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:25:32.462245 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-05dc2006-54cb-4f8d-97a3-6016a13270ab admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:32.463153 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-05dc2006-54cb-4f8d-97a3-6016a13270ab admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:32.465014 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 861/1743] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:25:32 2022] DELETE /identity/v3/projects/e8f0ca3a9ed74a2d847924798507e1cc => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:32.470521 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5624f6b-7138-48d0-887f-b2f37a17418d None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:32.471032 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5624f6b-7138-48d0-887f-b2f37a17418d None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:32.471132 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5624f6b-7138-48d0-887f-b2f37a17418d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:32.492704 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-a5624f6b-7138-48d0-887f-b2f37a17418d None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:32.494120 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-a5624f6b-7138-48d0-887f-b2f37a17418d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:32.516194 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:32.570322 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 883/1744] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:25:32.578317 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea9ad7c-cb5d-4a3f-aa0d-57bfa2c991cf None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:32.578655 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea9ad7c-cb5d-4a3f-aa0d-57bfa2c991cf None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:32.579529 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea9ad7c-cb5d-4a3f-aa0d-57bfa2c991cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:32.607453 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-5ea9ad7c-cb5d-4a3f-aa0d-57bfa2c991cf None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:32.608697 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-5ea9ad7c-cb5d-4a3f-aa0d-57bfa2c991cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:32.617727 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 862/1745] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:25:32.621842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5624f6b-7138-48d0-887f-b2f37a17418d None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:32.633756 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5624f6b-7138-48d0-887f-b2f37a17418d None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:32.638453 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b938da8-bbfa-4b7b-a246-be49e7b8cb33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:32.639409 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b938da8-bbfa-4b7b-a246-be49e7b8cb33 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:32.639690 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b938da8-bbfa-4b7b-a246-be49e7b8cb33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:32.639964 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b938da8-bbfa-4b7b-a246-be49e7b8cb33 admin admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:32.640656 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b938da8-bbfa-4b7b-a246-be49e7b8cb33 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:25:32.640957 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b938da8-bbfa-4b7b-a246-be49e7b8cb33 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:32.642637 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b938da8-bbfa-4b7b-a246-be49e7b8cb33 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:32.655829 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 884/1746] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:25:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:25:32.660297 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea9ad7c-cb5d-4a3f-aa0d-57bfa2c991cf None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:32.670020 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea9ad7c-cb5d-4a3f-aa0d-57bfa2c991cf None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:32.675417 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b03d11-1fea-49ee-96a4-98216121e9bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:32.676035 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b03d11-1fea-49ee-96a4-98216121e9bb admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:32.676841 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b03d11-1fea-49ee-96a4-98216121e9bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:32.677117 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b03d11-1fea-49ee-96a4-98216121e9bb admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:32.677944 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b03d11-1fea-49ee-96a4-98216121e9bb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:32.679931 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b03d11-1fea-49ee-96a4-98216121e9bb admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:32.712980 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 863/1747] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:25:32 2022] POST /identity/v3/projects => generated 387 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:25:32.714094 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b938da8-bbfa-4b7b-a246-be49e7b8cb33 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:32.724420 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b938da8-bbfa-4b7b-a246-be49e7b8cb33 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:32.794588 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72817d2b-064f-4c04-8aff-8018a24ed1bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:32.795157 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72817d2b-064f-4c04-8aff-8018a24ed1bd admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:32.795340 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72817d2b-064f-4c04-8aff-8018a24ed1bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:32.795704 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72817d2b-064f-4c04-8aff-8018a24ed1bd admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:32.797476 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72817d2b-064f-4c04-8aff-8018a24ed1bd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:32.799868 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72817d2b-064f-4c04-8aff-8018a24ed1bd admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:32.828884 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 885/1748] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:25:32 2022] POST /identity/v3/users => generated 294 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:25:32.832892 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b03d11-1fea-49ee-96a4-98216121e9bb admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:32.843629 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b03d11-1fea-49ee-96a4-98216121e9bb admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:32.848875 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf6f88b-2576-47c0-9196-ecf73261c9f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:32.849519 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf6f88b-2576-47c0-9196-ecf73261c9f2 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:32.849886 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf6f88b-2576-47c0-9196-ecf73261c9f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:32.850162 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf6f88b-2576-47c0-9196-ecf73261c9f2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:32.850804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf6f88b-2576-47c0-9196-ecf73261c9f2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:32.852399 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf6f88b-2576-47c0-9196-ecf73261c9f2 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:32.865660 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 864/1749] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:25:32 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 27 01:25:32.866915 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72817d2b-064f-4c04-8aff-8018a24ed1bd admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:32.878810 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72817d2b-064f-4c04-8aff-8018a24ed1bd admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:32.885451 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c68f22b9-5e65-49e5-887d-46173ca944cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:32.886184 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68f22b9-5e65-49e5-887d-46173ca944cd admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:32.886579 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68f22b9-5e65-49e5-887d-46173ca944cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:32.886922 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68f22b9-5e65-49e5-887d-46173ca944cd admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:32.887708 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c68f22b9-5e65-49e5-887d-46173ca944cd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:32.889571 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c68f22b9-5e65-49e5-887d-46173ca944cd admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:32.903161 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 886/1750] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:25:32 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:25:32.907019 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf6f88b-2576-47c0-9196-ecf73261c9f2 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:32.922164 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf6f88b-2576-47c0-9196-ecf73261c9f2 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:32.927759 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec180f1e-a49a-42cf-b30c-4731ac6bb80a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:32.928428 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec180f1e-a49a-42cf-b30c-4731ac6bb80a admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:32.928704 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec180f1e-a49a-42cf-b30c-4731ac6bb80a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:32.929694 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec180f1e-a49a-42cf-b30c-4731ac6bb80a admin admin] PATH_INFO: `/v3/projects/26e9a0ba8de8477592463173b3a67ae2/users/c922318ea6a04cfbbe1d29692eff4d1c/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:32.951618 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec180f1e-a49a-42cf-b30c-4731ac6bb80a admin admin] RBAC: Authorizing `identity:create_grant(project_id=26e9a0ba8de8477592463173b3a67ae2, user_id=c922318ea6a04cfbbe1d29692eff4d1c, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:32.955341 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec180f1e-a49a-42cf-b30c-4731ac6bb80a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:32.974169 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 865/1751] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:25:32 2022] PUT /identity/v3/projects/26e9a0ba8de8477592463173b3a67ae2/users/c922318ea6a04cfbbe1d29692eff4d1c/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:25:32.989115 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44eec5dd-363b-46a8-ac25-8a3994d8c007 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:32.989850 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44eec5dd-363b-46a8-ac25-8a3994d8c007 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:32.990234 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44eec5dd-363b-46a8-ac25-8a3994d8c007 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:33.036172 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-44eec5dd-363b-46a8-ac25-8a3994d8c007 None None] MFA Rules not processed for user `c922318ea6a04cfbbe1d29692eff4d1c`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:33.037716 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-44eec5dd-363b-46a8-ac25-8a3994d8c007 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:33.159681 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 887/1752] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:32 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:25:33.164444 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec180f1e-a49a-42cf-b30c-4731ac6bb80a admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:33.249622 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec180f1e-a49a-42cf-b30c-4731ac6bb80a admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:33.257169 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f7f4d2-1807-42ad-8efc-d30d86067718 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:33.258014 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f7f4d2-1807-42ad-8efc-d30d86067718 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:33.258407 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f7f4d2-1807-42ad-8efc-d30d86067718 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:33.258761 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f7f4d2-1807-42ad-8efc-d30d86067718 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:33.259900 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f7f4d2-1807-42ad-8efc-d30d86067718 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:33.261933 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f7f4d2-1807-42ad-8efc-d30d86067718 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:33.273765 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:33.304727 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 866/1753] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:25:33 2022] POST /identity/v3/projects => generated 385 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:25:33.309298 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44eec5dd-363b-46a8-ac25-8a3994d8c007 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:33.322372 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44eec5dd-363b-46a8-ac25-8a3994d8c007 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:33.336166 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:33.386249 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713fadbb-530e-4482-9a4e-2417d8cd5c50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:33.386969 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713fadbb-530e-4482-9a4e-2417d8cd5c50 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:33.387238 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713fadbb-530e-4482-9a4e-2417d8cd5c50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:33.387496 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713fadbb-530e-4482-9a4e-2417d8cd5c50 admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:33.388129 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713fadbb-530e-4482-9a4e-2417d8cd5c50 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:33.390169 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713fadbb-530e-4482-9a4e-2417d8cd5c50 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:33.416141 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 888/1754] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:25:33 2022] POST /identity/v3/users => generated 299 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:25:33.417418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f7f4d2-1807-42ad-8efc-d30d86067718 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:33.428708 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f7f4d2-1807-42ad-8efc-d30d86067718 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:33.434929 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58df590e-7906-4451-9576-36225eccd43c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:33.435656 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58df590e-7906-4451-9576-36225eccd43c admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:33.435836 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58df590e-7906-4451-9576-36225eccd43c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:33.436072 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58df590e-7906-4451-9576-36225eccd43c admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:33.436862 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58df590e-7906-4451-9576-36225eccd43c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:33.438669 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58df590e-7906-4451-9576-36225eccd43c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:33.450111 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 867/1755] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:25:33 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 27 01:25:33.454640 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713fadbb-530e-4482-9a4e-2417d8cd5c50 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:33.467334 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713fadbb-530e-4482-9a4e-2417d8cd5c50 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:33.474471 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e568f413-5574-41c0-a70c-285a6e03dfd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:33.475528 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e568f413-5574-41c0-a70c-285a6e03dfd0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:33.475805 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e568f413-5574-41c0-a70c-285a6e03dfd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:33.476075 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e568f413-5574-41c0-a70c-285a6e03dfd0 admin admin] PATH_INFO: `/v3/projects/f5f2855ff504424288df35d177ece2bf/users/edc3f520046841818770f3639cc8d596/roles/414286927fdd44688107ff5e10235acf` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:33.501372 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e568f413-5574-41c0-a70c-285a6e03dfd0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f5f2855ff504424288df35d177ece2bf, user_id=edc3f520046841818770f3639cc8d596, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:33.505662 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e568f413-5574-41c0-a70c-285a6e03dfd0 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:33.537608 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 889/1756] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:25:33 2022] PUT /identity/v3/projects/f5f2855ff504424288df35d177ece2bf/users/edc3f520046841818770f3639cc8d596/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:33.542620 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58df590e-7906-4451-9576-36225eccd43c admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:33.553008 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58df590e-7906-4451-9576-36225eccd43c admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:33.618804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd1e8d77-1125-4f96-b3c2-cfce4f978bf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:33.619422 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1e8d77-1125-4f96-b3c2-cfce4f978bf0 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:33.619699 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1e8d77-1125-4f96-b3c2-cfce4f978bf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:33.619969 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1e8d77-1125-4f96-b3c2-cfce4f978bf0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:33.621505 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd1e8d77-1125-4f96-b3c2-cfce4f978bf0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:33.625489 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd1e8d77-1125-4f96-b3c2-cfce4f978bf0 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:33.638024 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 868/1757] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:25:33 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:25:33.642104 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e568f413-5574-41c0-a70c-285a6e03dfd0 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:33.653363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e568f413-5574-41c0-a70c-285a6e03dfd0 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:33.659056 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68fde519-822b-414a-ab2d-2d6e72adfb01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:33.659831 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fde519-822b-414a-ab2d-2d6e72adfb01 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:33.660179 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fde519-822b-414a-ab2d-2d6e72adfb01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:33.660573 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fde519-822b-414a-ab2d-2d6e72adfb01 admin admin] PATH_INFO: `/v3/projects/f5f2855ff504424288df35d177ece2bf/users/edc3f520046841818770f3639cc8d596/roles/414286927fdd44688107ff5e10235acf` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:33.663550 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68fde519-822b-414a-ab2d-2d6e72adfb01 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f5f2855ff504424288df35d177ece2bf, user_id=edc3f520046841818770f3639cc8d596, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:33.666918 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68fde519-822b-414a-ab2d-2d6e72adfb01 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:33.680392 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 890/1758] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:25:33 2022] PUT /identity/v3/projects/f5f2855ff504424288df35d177ece2bf/users/edc3f520046841818770f3639cc8d596/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:33.692453 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c38368-1a58-41fa-8de9-4302fd1bb55f None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:33.692914 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c38368-1a58-41fa-8de9-4302fd1bb55f None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:33.694118 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c38368-1a58-41fa-8de9-4302fd1bb55f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:33.725461 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-72c38368-1a58-41fa-8de9-4302fd1bb55f None None] MFA Rules not processed for user `edc3f520046841818770f3639cc8d596`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:33.726824 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-72c38368-1a58-41fa-8de9-4302fd1bb55f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:33.827346 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 869/1759] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:33 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:25:33.839778 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68fde519-822b-414a-ab2d-2d6e72adfb01 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:33.841903 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-68fde519-822b-414a-ab2d-2d6e72adfb01 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:33.916251 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68fde519-822b-414a-ab2d-2d6e72adfb01 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:33.920410 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4024d5-6b5c-4373-931c-32a33ca036c6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:33.921021 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4024d5-6b5c-4373-931c-32a33ca036c6 service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:33.921419 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4024d5-6b5c-4373-931c-32a33ca036c6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:33.922034 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4024d5-6b5c-4373-931c-32a33ca036c6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:33.932921 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4024d5-6b5c-4373-931c-32a33ca036c6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:33.934462 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4024d5-6b5c-4373-931c-32a33ca036c6 service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:33.945555 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 891/1760] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:25:33 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:25:34.038644 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c38368-1a58-41fa-8de9-4302fd1bb55f None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:34.102451 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:34.107084 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c38368-1a58-41fa-8de9-4302fd1bb55f None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:34.113185 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b0c28d-898d-4c74-9ffd-f3c6f636acba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:34.113831 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b0c28d-898d-4c74-9ffd-f3c6f636acba admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:34.114108 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b0c28d-898d-4c74-9ffd-f3c6f636acba admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:34.114377 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b0c28d-898d-4c74-9ffd-f3c6f636acba admin admin] PATH_INFO: `/v3/users/c922318ea6a04cfbbe1d29692eff4d1c` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:34.115474 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12b0c28d-898d-4c74-9ffd-f3c6f636acba admin admin] RBAC: Authorizing `identity:delete_user(user_id=c922318ea6a04cfbbe1d29692eff4d1c)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:34.117432 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12b0c28d-898d-4c74-9ffd-f3c6f636acba admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:34.176991 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-12b0c28d-898d-4c74-9ffd-f3c6f636acba admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c922318ea6a04cfbbe1d29692eff4d1c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:34.185786 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-12b0c28d-898d-4c74-9ffd-f3c6f636acba admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c922318ea6a04cfbbe1d29692eff4d1c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:34.186429 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-12b0c28d-898d-4c74-9ffd-f3c6f636acba admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c922318ea6a04cfbbe1d29692eff4d1c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:34.198557 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-12b0c28d-898d-4c74-9ffd-f3c6f636acba admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c922318ea6a04cfbbe1d29692eff4d1c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:34.205226 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-12b0c28d-898d-4c74-9ffd-f3c6f636acba admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c922318ea6a04cfbbe1d29692eff4d1c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:34.211743 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 870/1761] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:25:34 2022] DELETE /identity/v3/users/c922318ea6a04cfbbe1d29692eff4d1c => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:34.216234 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70309f1e-9cc0-40f9-866f-cf1ce5b78f0d None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:34.216658 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70309f1e-9cc0-40f9-866f-cf1ce5b78f0d None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:34.216989 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70309f1e-9cc0-40f9-866f-cf1ce5b78f0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:34.235952 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-70309f1e-9cc0-40f9-866f-cf1ce5b78f0d None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:34.237266 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-70309f1e-9cc0-40f9-866f-cf1ce5b78f0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:34.299303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 892/1762] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:25:34.302212 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b0c28d-898d-4c74-9ffd-f3c6f636acba admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:34.309757 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-12b0c28d-898d-4c74-9ffd-f3c6f636acba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:34.309990 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70309f1e-9cc0-40f9-866f-cf1ce5b78f0d None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:34.311407 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-70309f1e-9cc0-40f9-866f-cf1ce5b78f0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:34.323866 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b0c28d-898d-4c74-9ffd-f3c6f636acba admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:34.329155 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e7be3a-4eb1-4176-a95f-6db841c1e014 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:34.329643 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:34.329774 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e7be3a-4eb1-4176-a95f-6db841c1e014 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:34.330063 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e7be3a-4eb1-4176-a95f-6db841c1e014 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:34.330332 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e7be3a-4eb1-4176-a95f-6db841c1e014 admin admin] PATH_INFO: `/v3/users/8d73612498fa4689aae23096ab5dfa1c` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:34.331665 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e7be3a-4eb1-4176-a95f-6db841c1e014 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8d73612498fa4689aae23096ab5dfa1c)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:34.334408 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e7be3a-4eb1-4176-a95f-6db841c1e014 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:34.377237 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70309f1e-9cc0-40f9-866f-cf1ce5b78f0d None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:34.381628 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f27d97-501a-4676-8993-36a442a35a56 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:34.382242 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f27d97-501a-4676-8993-36a442a35a56 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:34.382517 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f27d97-501a-4676-8993-36a442a35a56 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:34.382789 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f27d97-501a-4676-8993-36a442a35a56 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:34.384020 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-c0e7be3a-4eb1-4176-a95f-6db841c1e014 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8d73612498fa4689aae23096ab5dfa1c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:34.391798 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-c0e7be3a-4eb1-4176-a95f-6db841c1e014 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8d73612498fa4689aae23096ab5dfa1c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:34.392712 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-c0e7be3a-4eb1-4176-a95f-6db841c1e014 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8d73612498fa4689aae23096ab5dfa1c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:34.393238 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f27d97-501a-4676-8993-36a442a35a56 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:34.394668 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f27d97-501a-4676-8993-36a442a35a56 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:34.404958 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 893/1763] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:25:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 27 01:25:34.405977 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-c0e7be3a-4eb1-4176-a95f-6db841c1e014 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8d73612498fa4689aae23096ab5dfa1c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:34.413450 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-c0e7be3a-4eb1-4176-a95f-6db841c1e014 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8d73612498fa4689aae23096ab5dfa1c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:34.419520 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 871/1764] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:25:34 2022] DELETE /identity/v3/users/8d73612498fa4689aae23096ab5dfa1c => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:34.424110 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca605ab-5f6b-450b-b203-967aac299589 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:34.424391 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca605ab-5f6b-450b-b203-967aac299589 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:34.424661 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca605ab-5f6b-450b-b203-967aac299589 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:34.446601 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-bca605ab-5f6b-450b-b203-967aac299589 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:34.448016 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-bca605ab-5f6b-450b-b203-967aac299589 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:34.514011 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 894/1765] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:25:34.524013 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e7be3a-4eb1-4176-a95f-6db841c1e014 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:34.525473 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-c0e7be3a-4eb1-4176-a95f-6db841c1e014 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:34.586327 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:34.591273 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e7be3a-4eb1-4176-a95f-6db841c1e014 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:34.596043 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ecd897-782a-420e-b6cd-daae7f6360ad service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:34.596879 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ecd897-782a-420e-b6cd-daae7f6360ad service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:34.597343 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ecd897-782a-420e-b6cd-daae7f6360ad service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:34.597750 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ecd897-782a-420e-b6cd-daae7f6360ad service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:34.610972 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ecd897-782a-420e-b6cd-daae7f6360ad service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:34.612543 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ecd897-782a-420e-b6cd-daae7f6360ad service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:34.622548 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 872/1766] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:25:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 27 01:25:34.729782 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca605ab-5f6b-450b-b203-967aac299589 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:34.731231 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-bca605ab-5f6b-450b-b203-967aac299589 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:34.744797 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca605ab-5f6b-450b-b203-967aac299589 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:34.750458 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1b5bc5-b008-4ab0-9202-f0d52e73448e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:34.751399 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1b5bc5-b008-4ab0-9202-f0d52e73448e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:34.751750 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1b5bc5-b008-4ab0-9202-f0d52e73448e admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:34.752081 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1b5bc5-b008-4ab0-9202-f0d52e73448e admin admin] PATH_INFO: `/v3/projects/26e9a0ba8de8477592463173b3a67ae2` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:34.753407 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e1b5bc5-b008-4ab0-9202-f0d52e73448e admin admin] RBAC: Authorizing `identity:delete_project(project_id=26e9a0ba8de8477592463173b3a67ae2)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:34.756532 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e1b5bc5-b008-4ab0-9202-f0d52e73448e admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:34.807870 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-5e1b5bc5-b008-4ab0-9202-f0d52e73448e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '26e9a0ba8de8477592463173b3a67ae2', 'request_id': 'req-5e1b5bc5-b008-4ab0-9202-f0d52e73448e'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:34.831210 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-5e1b5bc5-b008-4ab0-9202-f0d52e73448e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '26e9a0ba8de8477592463173b3a67ae2', 'request_id': 'req-5e1b5bc5-b008-4ab0-9202-f0d52e73448e'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:34.839205 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-5e1b5bc5-b008-4ab0-9202-f0d52e73448e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '26e9a0ba8de8477592463173b3a67ae2', 'request_id': 'req-5e1b5bc5-b008-4ab0-9202-f0d52e73448e'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:34.849777 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-5e1b5bc5-b008-4ab0-9202-f0d52e73448e admin admin] The token cache is being invalidate because project 26e9a0ba8de8477592463173b3a67ae2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:25:34.850149 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-5e1b5bc5-b008-4ab0-9202-f0d52e73448e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:34.850874 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-5e1b5bc5-b008-4ab0-9202-f0d52e73448e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:34.852515 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 895/1767] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:25:34 2022] DELETE /identity/v3/projects/26e9a0ba8de8477592463173b3a67ae2 => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:34.856462 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ecd897-782a-420e-b6cd-daae7f6360ad service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:34.866795 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ecd897-782a-420e-b6cd-daae7f6360ad service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:34.924875 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0ed019-05c7-4374-a226-03c0cf33651b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:34.925785 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0ed019-05c7-4374-a226-03c0cf33651b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:34.926828 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0ed019-05c7-4374-a226-03c0cf33651b admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:34.927290 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0ed019-05c7-4374-a226-03c0cf33651b admin admin] PATH_INFO: `/v3/users/edc3f520046841818770f3639cc8d596` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:34.928909 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0ed019-05c7-4374-a226-03c0cf33651b admin admin] RBAC: Authorizing `identity:delete_user(user_id=edc3f520046841818770f3639cc8d596)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:34.931249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0ed019-05c7-4374-a226-03c0cf33651b admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:34.946717 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1b5bc5-b008-4ab0-9202-f0d52e73448e admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:34.948250 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-5e1b5bc5-b008-4ab0-9202-f0d52e73448e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:34.962628 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1b5bc5-b008-4ab0-9202-f0d52e73448e admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:34.969828 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937a42e1-07b3-47e4-b6f9-54da7ff91f49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:34.970456 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937a42e1-07b3-47e4-b6f9-54da7ff91f49 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:34.970872 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937a42e1-07b3-47e4-b6f9-54da7ff91f49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:34.971135 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937a42e1-07b3-47e4-b6f9-54da7ff91f49 admin admin] PATH_INFO: `/v3/projects/96f48dca6fc54975affc87292e52e111` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:34.972391 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-937a42e1-07b3-47e4-b6f9-54da7ff91f49 admin admin] RBAC: Authorizing `identity:delete_project(project_id=96f48dca6fc54975affc87292e52e111)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:34.975242 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-937a42e1-07b3-47e4-b6f9-54da7ff91f49 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:34.987130 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6d0ed019-05c7-4374-a226-03c0cf33651b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'edc3f520046841818770f3639cc8d596'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:34.995154 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6d0ed019-05c7-4374-a226-03c0cf33651b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'edc3f520046841818770f3639cc8d596'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:34.995807 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6d0ed019-05c7-4374-a226-03c0cf33651b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'edc3f520046841818770f3639cc8d596'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:35.009947 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6d0ed019-05c7-4374-a226-03c0cf33651b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'edc3f520046841818770f3639cc8d596'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:35.015531 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:35.017121 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6d0ed019-05c7-4374-a226-03c0cf33651b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'edc3f520046841818770f3639cc8d596'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:35.024727 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 873/1768] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:25:34 2022] DELETE /identity/v3/users/edc3f520046841818770f3639cc8d596 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:25:35.028420 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-937a42e1-07b3-47e4-b6f9-54da7ff91f49 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '96f48dca6fc54975affc87292e52e111', 'request_id': 'req-937a42e1-07b3-47e4-b6f9-54da7ff91f49'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:35.038195 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-937a42e1-07b3-47e4-b6f9-54da7ff91f49 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '96f48dca6fc54975affc87292e52e111', 'request_id': 'req-937a42e1-07b3-47e4-b6f9-54da7ff91f49'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:35.045442 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-937a42e1-07b3-47e4-b6f9-54da7ff91f49 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '96f48dca6fc54975affc87292e52e111', 'request_id': 'req-937a42e1-07b3-47e4-b6f9-54da7ff91f49'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:35.058185 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-937a42e1-07b3-47e4-b6f9-54da7ff91f49 admin admin] The token cache is being invalidate because project 96f48dca6fc54975affc87292e52e111 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:25:35.058471 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-937a42e1-07b3-47e4-b6f9-54da7ff91f49 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:35.060240 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-937a42e1-07b3-47e4-b6f9-54da7ff91f49 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:35.064596 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 896/1769] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:25:34 2022] DELETE /identity/v3/projects/96f48dca6fc54975affc87292e52e111 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:25:35.066312 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0ed019-05c7-4374-a226-03c0cf33651b admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:35.068728 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-6d0ed019-05c7-4374-a226-03c0cf33651b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:35.148664 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0ed019-05c7-4374-a226-03c0cf33651b admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:35.153379 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900feb6d-054a-408d-b6f1-b49d058c4627 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:35.154012 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900feb6d-054a-408d-b6f1-b49d058c4627 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:35.154285 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900feb6d-054a-408d-b6f1-b49d058c4627 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:35.154549 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900feb6d-054a-408d-b6f1-b49d058c4627 admin admin] PATH_INFO: `/v3/users/fa80d74e22fe4578af8f994818681496` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:35.155670 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-900feb6d-054a-408d-b6f1-b49d058c4627 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fa80d74e22fe4578af8f994818681496)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:35.157807 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-900feb6d-054a-408d-b6f1-b49d058c4627 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:35.171358 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:35.212319 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-900feb6d-054a-408d-b6f1-b49d058c4627 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fa80d74e22fe4578af8f994818681496'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:35.224016 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-900feb6d-054a-408d-b6f1-b49d058c4627 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fa80d74e22fe4578af8f994818681496'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:35.224876 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-900feb6d-054a-408d-b6f1-b49d058c4627 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fa80d74e22fe4578af8f994818681496'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:35.238771 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-900feb6d-054a-408d-b6f1-b49d058c4627 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fa80d74e22fe4578af8f994818681496'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:35.246594 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-900feb6d-054a-408d-b6f1-b49d058c4627 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fa80d74e22fe4578af8f994818681496'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:35.253240 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 874/1770] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:25:35 2022] DELETE /identity/v3/users/fa80d74e22fe4578af8f994818681496 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:35.615170 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937a42e1-07b3-47e4-b6f9-54da7ff91f49 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:35.616660 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-937a42e1-07b3-47e4-b6f9-54da7ff91f49 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:35.666084 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900feb6d-054a-408d-b6f1-b49d058c4627 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:35.667624 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-900feb6d-054a-408d-b6f1-b49d058c4627 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:35.690311 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937a42e1-07b3-47e4-b6f9-54da7ff91f49 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:35.755667 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900feb6d-054a-408d-b6f1-b49d058c4627 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:35.760660 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff2f21b-d3c2-4b1d-9814-dc1ac988b1c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:35.761371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff2f21b-d3c2-4b1d-9814-dc1ac988b1c8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:35.761652 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff2f21b-d3c2-4b1d-9814-dc1ac988b1c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:35.762009 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff2f21b-d3c2-4b1d-9814-dc1ac988b1c8 admin admin] PATH_INFO: `/v3/projects/d28bf017913d46819ed954f74d8d1d34` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:35.763049 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff2f21b-d3c2-4b1d-9814-dc1ac988b1c8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d28bf017913d46819ed954f74d8d1d34)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:35.763598 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d43bdd7-5dc6-4ad8-bdb5-731735e36969 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:35.764269 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d43bdd7-5dc6-4ad8-bdb5-731735e36969 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:35.764547 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d43bdd7-5dc6-4ad8-bdb5-731735e36969 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:35.764819 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d43bdd7-5dc6-4ad8-bdb5-731735e36969 admin admin] PATH_INFO: `/v3/projects/f5f2855ff504424288df35d177ece2bf` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:35.764976 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff2f21b-d3c2-4b1d-9814-dc1ac988b1c8 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:35.765990 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d43bdd7-5dc6-4ad8-bdb5-731735e36969 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f5f2855ff504424288df35d177ece2bf)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:35.767964 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d43bdd7-5dc6-4ad8-bdb5-731735e36969 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:35.818076 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-dff2f21b-d3c2-4b1d-9814-dc1ac988b1c8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd28bf017913d46819ed954f74d8d1d34', 'request_id': 'req-dff2f21b-d3c2-4b1d-9814-dc1ac988b1c8'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:35.826214 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-0d43bdd7-5dc6-4ad8-bdb5-731735e36969 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f5f2855ff504424288df35d177ece2bf', 'request_id': 'req-0d43bdd7-5dc6-4ad8-bdb5-731735e36969'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:35.827860 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-dff2f21b-d3c2-4b1d-9814-dc1ac988b1c8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd28bf017913d46819ed954f74d8d1d34', 'request_id': 'req-dff2f21b-d3c2-4b1d-9814-dc1ac988b1c8'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:35.835104 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-0d43bdd7-5dc6-4ad8-bdb5-731735e36969 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f5f2855ff504424288df35d177ece2bf', 'request_id': 'req-0d43bdd7-5dc6-4ad8-bdb5-731735e36969'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:35.835846 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-dff2f21b-d3c2-4b1d-9814-dc1ac988b1c8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd28bf017913d46819ed954f74d8d1d34', 'request_id': 'req-dff2f21b-d3c2-4b1d-9814-dc1ac988b1c8'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:35.841980 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-0d43bdd7-5dc6-4ad8-bdb5-731735e36969 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f5f2855ff504424288df35d177ece2bf', 'request_id': 'req-0d43bdd7-5dc6-4ad8-bdb5-731735e36969'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:35.847492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-dff2f21b-d3c2-4b1d-9814-dc1ac988b1c8 admin admin] The token cache is being invalidate because project d28bf017913d46819ed954f74d8d1d34 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:25:35.847853 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-dff2f21b-d3c2-4b1d-9814-dc1ac988b1c8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:35.849028 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-dff2f21b-d3c2-4b1d-9814-dc1ac988b1c8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:35.850405 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:DELETE=12,keystone:SELECT=86,keystone:INSERT=2 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:35.850672 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 875/1771] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:25:35 2022] DELETE /identity/v3/projects/d28bf017913d46819ed954f74d8d1d34 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:35.854743 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-0d43bdd7-5dc6-4ad8-bdb5-731735e36969 admin admin] The token cache is being invalidate because project f5f2855ff504424288df35d177ece2bf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:25:35.855216 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-0d43bdd7-5dc6-4ad8-bdb5-731735e36969 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:35.855849 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ba72cb-93d8-4d07-81ef-608139fa4958 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:35.855991 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ba72cb-93d8-4d07-81ef-608139fa4958 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:35.856209 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ba72cb-93d8-4d07-81ef-608139fa4958 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:35.856471 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-0d43bdd7-5dc6-4ad8-bdb5-731735e36969 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:35.858444 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 897/1772] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:25:35 2022] DELETE /identity/v3/projects/f5f2855ff504424288df35d177ece2bf => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:35.864588 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb248777-dcbe-47c4-8c65-7b0940e0281e None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:35.864944 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb248777-dcbe-47c4-8c65-7b0940e0281e None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:35.865320 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb248777-dcbe-47c4-8c65-7b0940e0281e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:35.876794 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-90ba72cb-93d8-4d07-81ef-608139fa4958 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:35.878093 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-90ba72cb-93d8-4d07-81ef-608139fa4958 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:35.885449 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-eb248777-dcbe-47c4-8c65-7b0940e0281e None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:35.886676 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-eb248777-dcbe-47c4-8c65-7b0940e0281e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:35.891351 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:35.950781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 876/1773] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:25:35.958334 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77db8dde-be68-4385-847d-07f2cb9623d1 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:35.958697 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77db8dde-be68-4385-847d-07f2cb9623d1 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:35.958970 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77db8dde-be68-4385-847d-07f2cb9623d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:35.959231 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 898/1774] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:25:35.966496 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abaec91-5e1d-4ef6-aac6-f45c8a6a63e8 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:35.966855 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abaec91-5e1d-4ef6-aac6-f45c8a6a63e8 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:35.967195 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abaec91-5e1d-4ef6-aac6-f45c8a6a63e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:35.985441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-77db8dde-be68-4385-847d-07f2cb9623d1 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:35.986663 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-77db8dde-be68-4385-847d-07f2cb9623d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:35.988178 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-0abaec91-5e1d-4ef6-aac6-f45c8a6a63e8 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:35.990216 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-0abaec91-5e1d-4ef6-aac6-f45c8a6a63e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:35.996441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 877/1775] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:25:35.998214 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 899/1776] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:25:36.000445 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77db8dde-be68-4385-847d-07f2cb9623d1 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:36.003424 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0abaec91-5e1d-4ef6-aac6-f45c8a6a63e8 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:36.009837 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77db8dde-be68-4385-847d-07f2cb9623d1 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:36.013666 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0abaec91-5e1d-4ef6-aac6-f45c8a6a63e8 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:36.014541 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a5973b-a17c-4660-b69f-022973b82738 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:36.015142 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a5973b-a17c-4660-b69f-022973b82738 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:36.015418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a5973b-a17c-4660-b69f-022973b82738 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:36.015688 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a5973b-a17c-4660-b69f-022973b82738 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:36.016537 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9a5973b-a17c-4660-b69f-022973b82738 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:25:36.016847 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9a5973b-a17c-4660-b69f-022973b82738 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:36.019035 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5fb3f5-f08b-4aa3-891b-a931d7fcfc7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:36.019538 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9a5973b-a17c-4660-b69f-022973b82738 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:36.019736 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5fb3f5-f08b-4aa3-891b-a931d7fcfc7d admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:36.020375 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5fb3f5-f08b-4aa3-891b-a931d7fcfc7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:36.020854 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5fb3f5-f08b-4aa3-891b-a931d7fcfc7d admin admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:36.021627 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c5fb3f5-f08b-4aa3-891b-a931d7fcfc7d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:25:36.021996 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c5fb3f5-f08b-4aa3-891b-a931d7fcfc7d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:36.023806 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c5fb3f5-f08b-4aa3-891b-a931d7fcfc7d admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:36.033445 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 878/1777] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:25:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:25:36.037821 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 900/1778] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:25:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:25:36.038081 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a5973b-a17c-4660-b69f-022973b82738 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:36.041685 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5fb3f5-f08b-4aa3-891b-a931d7fcfc7d admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:36.049535 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a5973b-a17c-4660-b69f-022973b82738 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:36.052179 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5fb3f5-f08b-4aa3-891b-a931d7fcfc7d admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:36.054638 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d0fd49-4c0b-417a-ba2d-e064b9eb6960 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:36.055487 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d0fd49-4c0b-417a-ba2d-e064b9eb6960 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:36.055766 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d0fd49-4c0b-417a-ba2d-e064b9eb6960 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:36.056044 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d0fd49-4c0b-417a-ba2d-e064b9eb6960 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:36.056666 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34d0fd49-4c0b-417a-ba2d-e064b9eb6960 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:36.058059 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf97cc3b-3f7a-40f6-b7c5-2620645cae8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:36.058601 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34d0fd49-4c0b-417a-ba2d-e064b9eb6960 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:36.058874 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf97cc3b-3f7a-40f6-b7c5-2620645cae8c admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:36.059251 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf97cc3b-3f7a-40f6-b7c5-2620645cae8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:36.059622 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf97cc3b-3f7a-40f6-b7c5-2620645cae8c admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:36.060352 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf97cc3b-3f7a-40f6-b7c5-2620645cae8c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:36.062380 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf97cc3b-3f7a-40f6-b7c5-2620645cae8c admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:36.111269 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 901/1779] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:25:36 2022] POST /identity/v3/projects => generated 381 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:25:36.111466 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 879/1780] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:25:36 2022] POST /identity/v3/projects => generated 373 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:25:36.111561 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf97cc3b-3f7a-40f6-b7c5-2620645cae8c admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:36.114147 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d0fd49-4c0b-417a-ba2d-e064b9eb6960 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:36.174663 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf97cc3b-3f7a-40f6-b7c5-2620645cae8c admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:36.180280 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db23990-d76a-4cd6-9dd6-1998ed4e9ab8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:36.180895 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db23990-d76a-4cd6-9dd6-1998ed4e9ab8 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:36.181206 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db23990-d76a-4cd6-9dd6-1998ed4e9ab8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:36.181481 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db23990-d76a-4cd6-9dd6-1998ed4e9ab8 admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:36.181821 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d0fd49-4c0b-417a-ba2d-e064b9eb6960 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:36.182111 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db23990-d76a-4cd6-9dd6-1998ed4e9ab8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:36.184024 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db23990-d76a-4cd6-9dd6-1998ed4e9ab8 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:36.186914 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98cdb695-cc4e-4f69-a541-0225e213fed9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:36.187589 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98cdb695-cc4e-4f69-a541-0225e213fed9 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:36.187943 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98cdb695-cc4e-4f69-a541-0225e213fed9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:36.188288 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98cdb695-cc4e-4f69-a541-0225e213fed9 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:36.188986 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98cdb695-cc4e-4f69-a541-0225e213fed9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:36.190983 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98cdb695-cc4e-4f69-a541-0225e213fed9 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:36.212851 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 902/1781] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:25:36 2022] POST /identity/v3/users => generated 291 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:25:36.214225 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db23990-d76a-4cd6-9dd6-1998ed4e9ab8 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:36.216500 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 880/1782] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:25:36 2022] POST /identity/v3/users => generated 287 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:25:36.220273 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98cdb695-cc4e-4f69-a541-0225e213fed9 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:36.224193 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db23990-d76a-4cd6-9dd6-1998ed4e9ab8 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:36.228814 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ddd844e-b09f-4726-8c62-06dc73fea1a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:36.229434 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ddd844e-b09f-4726-8c62-06dc73fea1a0 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:36.229713 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ddd844e-b09f-4726-8c62-06dc73fea1a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:36.229926 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98cdb695-cc4e-4f69-a541-0225e213fed9 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:36.230032 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ddd844e-b09f-4726-8c62-06dc73fea1a0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:36.230730 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ddd844e-b09f-4726-8c62-06dc73fea1a0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:36.232341 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ddd844e-b09f-4726-8c62-06dc73fea1a0 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:36.234609 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74167c70-366b-4461-a680-d957bdc014b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:36.235197 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74167c70-366b-4461-a680-d957bdc014b3 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:36.235478 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74167c70-366b-4461-a680-d957bdc014b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:36.235749 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74167c70-366b-4461-a680-d957bdc014b3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:36.236388 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74167c70-366b-4461-a680-d957bdc014b3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:36.238064 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74167c70-366b-4461-a680-d957bdc014b3 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:36.242818 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 903/1783] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:25:36 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:25:36.246786 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ddd844e-b09f-4726-8c62-06dc73fea1a0 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:36.249770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 881/1784] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:25:36 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 27 01:25:36.253303 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74167c70-366b-4461-a680-d957bdc014b3 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:36.257205 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ddd844e-b09f-4726-8c62-06dc73fea1a0 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:36.263120 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee12d7c7-014a-4392-8a46-83a6e6ad1207 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:36.263870 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee12d7c7-014a-4392-8a46-83a6e6ad1207 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:36.264225 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee12d7c7-014a-4392-8a46-83a6e6ad1207 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:36.264336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74167c70-366b-4461-a680-d957bdc014b3 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:36.264568 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee12d7c7-014a-4392-8a46-83a6e6ad1207 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:36.265312 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee12d7c7-014a-4392-8a46-83a6e6ad1207 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:36.266970 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee12d7c7-014a-4392-8a46-83a6e6ad1207 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:36.269402 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d4781d-3351-402f-a34d-7570e321a752 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:36.270781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d4781d-3351-402f-a34d-7570e321a752 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:36.271015 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d4781d-3351-402f-a34d-7570e321a752 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:36.271393 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d4781d-3351-402f-a34d-7570e321a752 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:36.271909 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9d4781d-3351-402f-a34d-7570e321a752 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:36.273518 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9d4781d-3351-402f-a34d-7570e321a752 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:36.281243 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 904/1785] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:25:36 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 27 01:25:36.285272 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee12d7c7-014a-4392-8a46-83a6e6ad1207 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:36.286679 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 882/1786] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:25:36 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 27 01:25:36.290449 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d4781d-3351-402f-a34d-7570e321a752 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:36.300352 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d4781d-3351-402f-a34d-7570e321a752 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:36.306184 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f90edf-e40f-4411-a172-07c34e1e79eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:36.306862 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f90edf-e40f-4411-a172-07c34e1e79eb admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:36.307160 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f90edf-e40f-4411-a172-07c34e1e79eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:36.307424 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f90edf-e40f-4411-a172-07c34e1e79eb admin admin] PATH_INFO: `/v3/projects/452958c488164ad1ad36e13b7387cbc1/users/e1c00aa46dac42f4a330cb3a6e871735/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:36.310599 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee12d7c7-014a-4392-8a46-83a6e6ad1207 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:36.315533 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e1974c-9320-4cab-97e7-da55e236a3a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:36.316131 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e1974c-9320-4cab-97e7-da55e236a3a5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:36.316407 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e1974c-9320-4cab-97e7-da55e236a3a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:36.316676 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e1974c-9320-4cab-97e7-da55e236a3a5 admin admin] PATH_INFO: `/v3/projects/0b8faa47fe814da9b362b5035f8eb2e5/users/f70044643e494cfa8d66f80a023bfd0c/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:36.324338 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97f90edf-e40f-4411-a172-07c34e1e79eb admin admin] RBAC: Authorizing `identity:create_grant(project_id=452958c488164ad1ad36e13b7387cbc1, user_id=e1c00aa46dac42f4a330cb3a6e871735, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:36.328091 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97f90edf-e40f-4411-a172-07c34e1e79eb admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:36.334424 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81e1974c-9320-4cab-97e7-da55e236a3a5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0b8faa47fe814da9b362b5035f8eb2e5, user_id=f70044643e494cfa8d66f80a023bfd0c, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:36.337453 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81e1974c-9320-4cab-97e7-da55e236a3a5 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:36.342762 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 883/1787] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:25:36 2022] PUT /identity/v3/projects/452958c488164ad1ad36e13b7387cbc1/users/e1c00aa46dac42f4a330cb3a6e871735/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:25:36.353197 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 905/1788] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:25:36 2022] PUT /identity/v3/projects/0b8faa47fe814da9b362b5035f8eb2e5/users/f70044643e494cfa8d66f80a023bfd0c/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:36.357981 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd684e76-7199-48ef-a64f-5187addc4ca9 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:36.358367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd684e76-7199-48ef-a64f-5187addc4ca9 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:36.358774 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd684e76-7199-48ef-a64f-5187addc4ca9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:36.363256 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c92bf17-6c2f-4ba3-8433-a46d673d7fa4 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:36.364440 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c92bf17-6c2f-4ba3-8433-a46d673d7fa4 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:36.364888 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c92bf17-6c2f-4ba3-8433-a46d673d7fa4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:36.374460 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:36.393582 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-bd684e76-7199-48ef-a64f-5187addc4ca9 None None] MFA Rules not processed for user `e1c00aa46dac42f4a330cb3a6e871735`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:36.395257 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-bd684e76-7199-48ef-a64f-5187addc4ca9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:36.404979 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-5c92bf17-6c2f-4ba3-8433-a46d673d7fa4 None None] MFA Rules not processed for user `f70044643e494cfa8d66f80a023bfd0c`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:36.405471 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:36.407029 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-5c92bf17-6c2f-4ba3-8433-a46d673d7fa4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:36.500397 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 884/1789] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:36 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:25:36.505705 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 906/1790] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:36 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:25:36.506229 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd684e76-7199-48ef-a64f-5187addc4ca9 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:36.514871 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c92bf17-6c2f-4ba3-8433-a46d673d7fa4 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:36.516306 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-5c92bf17-6c2f-4ba3-8433-a46d673d7fa4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:36.587729 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd684e76-7199-48ef-a64f-5187addc4ca9 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:36.591270 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c92bf17-6c2f-4ba3-8433-a46d673d7fa4 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:36.592985 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c02c8f-74ee-40b7-a3cd-3cdde7431557 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:36.593625 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c02c8f-74ee-40b7-a3cd-3cdde7431557 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:36.593909 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c02c8f-74ee-40b7-a3cd-3cdde7431557 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:36.594181 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c02c8f-74ee-40b7-a3cd-3cdde7431557 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:36.594870 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7c02c8f-74ee-40b7-a3cd-3cdde7431557 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:36.595593 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e88d026-210f-4aed-a593-9a6252052771 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:36.596294 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e88d026-210f-4aed-a593-9a6252052771 service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:36.596790 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7c02c8f-74ee-40b7-a3cd-3cdde7431557 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:36.596967 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e88d026-210f-4aed-a593-9a6252052771 service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:36.597373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e88d026-210f-4aed-a593-9a6252052771 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:36.607917 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e88d026-210f-4aed-a593-9a6252052771 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:36.609467 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e88d026-210f-4aed-a593-9a6252052771 service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:36.622330 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 907/1791] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:25:36 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 27 01:25:36.629568 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 885/1792] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:25:36 2022] POST /identity/v3/projects => generated 381 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:25:36.633631 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e88d026-210f-4aed-a593-9a6252052771 service cinder] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:36.645335 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e88d026-210f-4aed-a593-9a6252052771 service cinder] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:36.726309 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1545b39-fd59-4b6e-80da-efd9ead906e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:36.727614 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1545b39-fd59-4b6e-80da-efd9ead906e4 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:36.728044 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1545b39-fd59-4b6e-80da-efd9ead906e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:36.728507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1545b39-fd59-4b6e-80da-efd9ead906e4 admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:36.729459 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1545b39-fd59-4b6e-80da-efd9ead906e4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:36.731784 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1545b39-fd59-4b6e-80da-efd9ead906e4 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:36.763666 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 908/1793] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:25:36 2022] POST /identity/v3/users => generated 297 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:25:36.767985 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c02c8f-74ee-40b7-a3cd-3cdde7431557 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:36.779899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c02c8f-74ee-40b7-a3cd-3cdde7431557 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:36.784890 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fc87fa-6ed7-4ac0-a1f8-2bd36ac500cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:36.786051 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fc87fa-6ed7-4ac0-a1f8-2bd36ac500cb admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:36.786339 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fc87fa-6ed7-4ac0-a1f8-2bd36ac500cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:36.786624 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fc87fa-6ed7-4ac0-a1f8-2bd36ac500cb admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:36.787254 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47fc87fa-6ed7-4ac0-a1f8-2bd36ac500cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:36.788850 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47fc87fa-6ed7-4ac0-a1f8-2bd36ac500cb admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:36.801174 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 886/1794] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:25:36 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:25:36.804267 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1545b39-fd59-4b6e-80da-efd9ead906e4 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:36.815030 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1545b39-fd59-4b6e-80da-efd9ead906e4 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:36.820857 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23edf877-0b4f-488e-9e11-905ed98fdea2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:36.821863 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23edf877-0b4f-488e-9e11-905ed98fdea2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:36.822181 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23edf877-0b4f-488e-9e11-905ed98fdea2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:36.822426 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23edf877-0b4f-488e-9e11-905ed98fdea2 admin admin] PATH_INFO: `/v3/projects/755c3e1da4fa473ba51d8ae01f8faa87/users/1f8219f922c344c1997c2aeaec20623d/roles/414286927fdd44688107ff5e10235acf` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:36.841745 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23edf877-0b4f-488e-9e11-905ed98fdea2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=755c3e1da4fa473ba51d8ae01f8faa87, user_id=1f8219f922c344c1997c2aeaec20623d, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:36.845175 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23edf877-0b4f-488e-9e11-905ed98fdea2 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:36.856700 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:36.864980 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 909/1795] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:25:36 2022] PUT /identity/v3/projects/755c3e1da4fa473ba51d8ae01f8faa87/users/1f8219f922c344c1997c2aeaec20623d/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:36.868735 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fc87fa-6ed7-4ac0-a1f8-2bd36ac500cb admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:36.879258 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fc87fa-6ed7-4ac0-a1f8-2bd36ac500cb admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:36.955806 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2d026c-1acf-4e80-9675-7eadcf207650 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:36.956594 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2d026c-1acf-4e80-9675-7eadcf207650 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:36.957076 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2d026c-1acf-4e80-9675-7eadcf207650 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:36.958115 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2d026c-1acf-4e80-9675-7eadcf207650 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:36.958880 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2d026c-1acf-4e80-9675-7eadcf207650 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:36.960725 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2d026c-1acf-4e80-9675-7eadcf207650 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:36.981882 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 887/1796] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:25:36 2022] GET /identity/v3/roles => generated 1420 bytes in 114 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:25:36.986237 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23edf877-0b4f-488e-9e11-905ed98fdea2 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:36.999104 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23edf877-0b4f-488e-9e11-905ed98fdea2 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:37.007394 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24746fd0-3972-4b77-a56f-cfa4c02b9e16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:37.008339 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24746fd0-3972-4b77-a56f-cfa4c02b9e16 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:37.008693 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24746fd0-3972-4b77-a56f-cfa4c02b9e16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:37.009148 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24746fd0-3972-4b77-a56f-cfa4c02b9e16 admin admin] PATH_INFO: `/v3/projects/755c3e1da4fa473ba51d8ae01f8faa87/users/1f8219f922c344c1997c2aeaec20623d/roles/414286927fdd44688107ff5e10235acf` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:37.011577 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24746fd0-3972-4b77-a56f-cfa4c02b9e16 admin admin] RBAC: Authorizing `identity:create_grant(project_id=755c3e1da4fa473ba51d8ae01f8faa87, user_id=1f8219f922c344c1997c2aeaec20623d, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:37.014989 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24746fd0-3972-4b77-a56f-cfa4c02b9e16 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:37.033091 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 910/1797] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:25:36 2022] PUT /identity/v3/projects/755c3e1da4fa473ba51d8ae01f8faa87/users/1f8219f922c344c1997c2aeaec20623d/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:37.040761 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eaf721b-f611-4ebd-ad0f-775df0ff8f1d None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:37.041086 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eaf721b-f611-4ebd-ad0f-775df0ff8f1d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:37.041417 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eaf721b-f611-4ebd-ad0f-775df0ff8f1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:37.079907 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-8eaf721b-f611-4ebd-ad0f-775df0ff8f1d None None] MFA Rules not processed for user `1f8219f922c344c1997c2aeaec20623d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:37.081162 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-8eaf721b-f611-4ebd-ad0f-775df0ff8f1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:37.092751 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:37.158850 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 888/1798] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:37 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:25:37.167496 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24746fd0-3972-4b77-a56f-cfa4c02b9e16 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:37.226091 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24746fd0-3972-4b77-a56f-cfa4c02b9e16 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:37.229655 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42229680-8e52-48be-9203-c4006ebdb6c6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:37.230257 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42229680-8e52-48be-9203-c4006ebdb6c6 service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:37.230530 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42229680-8e52-48be-9203-c4006ebdb6c6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:37.230795 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42229680-8e52-48be-9203-c4006ebdb6c6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:37.240991 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42229680-8e52-48be-9203-c4006ebdb6c6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:37.242558 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42229680-8e52-48be-9203-c4006ebdb6c6 service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:37.252168 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 911/1799] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:25:37 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:25:37.365511 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eaf721b-f611-4ebd-ad0f-775df0ff8f1d None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:37.366914 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-8eaf721b-f611-4ebd-ad0f-775df0ff8f1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:37.378790 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eaf721b-f611-4ebd-ad0f-775df0ff8f1d None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:37.382660 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1433862-5aca-4e9d-8937-7a7829b1549c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:37.383338 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1433862-5aca-4e9d-8937-7a7829b1549c service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:37.383686 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1433862-5aca-4e9d-8937-7a7829b1549c service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:37.384908 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1433862-5aca-4e9d-8937-7a7829b1549c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:37.454717 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1433862-5aca-4e9d-8937-7a7829b1549c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:37.456151 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1433862-5aca-4e9d-8937-7a7829b1549c service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:37.465532 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 889/1800] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:25:37 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:25:45.385936 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42229680-8e52-48be-9203-c4006ebdb6c6 service cinder] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:45.451730 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42229680-8e52-48be-9203-c4006ebdb6c6 service cinder] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:45.456342 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba28bcae-ceb6-4131-9035-7663fe1292f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:45.456968 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba28bcae-ceb6-4131-9035-7663fe1292f5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:45.457779 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba28bcae-ceb6-4131-9035-7663fe1292f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:45.458801 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba28bcae-ceb6-4131-9035-7663fe1292f5 admin admin] PATH_INFO: `/v3/users/f70044643e494cfa8d66f80a023bfd0c` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:45.460064 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba28bcae-ceb6-4131-9035-7663fe1292f5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f70044643e494cfa8d66f80a023bfd0c)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:45.462216 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba28bcae-ceb6-4131-9035-7663fe1292f5 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:45.522927 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-ba28bcae-ceb6-4131-9035-7663fe1292f5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f70044643e494cfa8d66f80a023bfd0c'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:45.531824 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-ba28bcae-ceb6-4131-9035-7663fe1292f5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f70044643e494cfa8d66f80a023bfd0c'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:45.545624 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-ba28bcae-ceb6-4131-9035-7663fe1292f5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f70044643e494cfa8d66f80a023bfd0c'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:45.546879 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-ba28bcae-ceb6-4131-9035-7663fe1292f5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f70044643e494cfa8d66f80a023bfd0c'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:45.547398 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-ba28bcae-ceb6-4131-9035-7663fe1292f5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f70044643e494cfa8d66f80a023bfd0c'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:45.561370 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 912/1801] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:25:45 2022] DELETE /identity/v3/users/f70044643e494cfa8d66f80a023bfd0c => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:45.565802 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a30a87-0fea-4424-b180-040cea459bfd None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:45.566339 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a30a87-0fea-4424-b180-040cea459bfd None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:45.567332 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a30a87-0fea-4424-b180-040cea459bfd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:45.589113 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-36a30a87-0fea-4424-b180-040cea459bfd None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:45.589894 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-36a30a87-0fea-4424-b180-040cea459bfd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:45.667433 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 890/1802] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:25:45.674475 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba28bcae-ceb6-4131-9035-7663fe1292f5 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:45.676546 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-ba28bcae-ceb6-4131-9035-7663fe1292f5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:45.691712 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:45.742318 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba28bcae-ceb6-4131-9035-7663fe1292f5 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:45.746237 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2385b0e1-0c5c-4e2a-ad4d-73ca98329132 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:45.746838 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2385b0e1-0c5c-4e2a-ad4d-73ca98329132 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:45.747113 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2385b0e1-0c5c-4e2a-ad4d-73ca98329132 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:45.747378 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2385b0e1-0c5c-4e2a-ad4d-73ca98329132 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:45.756867 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2385b0e1-0c5c-4e2a-ad4d-73ca98329132 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:45.758302 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2385b0e1-0c5c-4e2a-ad4d-73ca98329132 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:45.768004 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 913/1803] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:25:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 27 01:25:46.053271 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a30a87-0fea-4424-b180-040cea459bfd None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:46.054789 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-36a30a87-0fea-4424-b180-040cea459bfd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:46.102489 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a30a87-0fea-4424-b180-040cea459bfd None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:46.108780 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8922c3b-b2c1-4eba-b2d2-f1dab250e58d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:46.109965 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8922c3b-b2c1-4eba-b2d2-f1dab250e58d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:46.110173 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8922c3b-b2c1-4eba-b2d2-f1dab250e58d admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:46.110449 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8922c3b-b2c1-4eba-b2d2-f1dab250e58d admin admin] PATH_INFO: `/v3/projects/0b8faa47fe814da9b362b5035f8eb2e5` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:46.111939 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8922c3b-b2c1-4eba-b2d2-f1dab250e58d admin admin] RBAC: Authorizing `identity:delete_project(project_id=0b8faa47fe814da9b362b5035f8eb2e5)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:46.115038 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8922c3b-b2c1-4eba-b2d2-f1dab250e58d admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:46.162249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:46.168849 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b8922c3b-b2c1-4eba-b2d2-f1dab250e58d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0b8faa47fe814da9b362b5035f8eb2e5', 'request_id': 'req-b8922c3b-b2c1-4eba-b2d2-f1dab250e58d'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:46.179377 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b8922c3b-b2c1-4eba-b2d2-f1dab250e58d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0b8faa47fe814da9b362b5035f8eb2e5', 'request_id': 'req-b8922c3b-b2c1-4eba-b2d2-f1dab250e58d'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:46.186192 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b8922c3b-b2c1-4eba-b2d2-f1dab250e58d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0b8faa47fe814da9b362b5035f8eb2e5', 'request_id': 'req-b8922c3b-b2c1-4eba-b2d2-f1dab250e58d'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:46.201184 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b8922c3b-b2c1-4eba-b2d2-f1dab250e58d admin admin] The token cache is being invalidate because project 0b8faa47fe814da9b362b5035f8eb2e5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:25:46.201184 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b8922c3b-b2c1-4eba-b2d2-f1dab250e58d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:46.201184 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b8922c3b-b2c1-4eba-b2d2-f1dab250e58d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:46.201567 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 891/1804] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:25:46 2022] DELETE /identity/v3/projects/0b8faa47fe814da9b362b5035f8eb2e5 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:46.205601 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2385b0e1-0c5c-4e2a-ad4d-73ca98329132 service neutron] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:46.216229 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2385b0e1-0c5c-4e2a-ad4d-73ca98329132 service neutron] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:46.273170 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68a5611-4e80-4005-b62b-19040b7e8274 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:46.273894 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68a5611-4e80-4005-b62b-19040b7e8274 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:46.274061 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68a5611-4e80-4005-b62b-19040b7e8274 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:46.274330 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68a5611-4e80-4005-b62b-19040b7e8274 admin admin] PATH_INFO: `/v3/users/1f8219f922c344c1997c2aeaec20623d` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:46.275517 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68a5611-4e80-4005-b62b-19040b7e8274 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1f8219f922c344c1997c2aeaec20623d)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:46.277457 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68a5611-4e80-4005-b62b-19040b7e8274 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:46.324491 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-b68a5611-4e80-4005-b62b-19040b7e8274 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1f8219f922c344c1997c2aeaec20623d'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:46.331551 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-b68a5611-4e80-4005-b62b-19040b7e8274 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1f8219f922c344c1997c2aeaec20623d'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:46.343602 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-b68a5611-4e80-4005-b62b-19040b7e8274 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1f8219f922c344c1997c2aeaec20623d'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:46.344161 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-b68a5611-4e80-4005-b62b-19040b7e8274 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1f8219f922c344c1997c2aeaec20623d'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:46.344756 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-b68a5611-4e80-4005-b62b-19040b7e8274 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1f8219f922c344c1997c2aeaec20623d'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:46.356548 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 914/1805] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:25:46 2022] DELETE /identity/v3/users/1f8219f922c344c1997c2aeaec20623d => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:46.625641 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8922c3b-b2c1-4eba-b2d2-f1dab250e58d admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:46.627066 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-b8922c3b-b2c1-4eba-b2d2-f1dab250e58d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:46.640074 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8922c3b-b2c1-4eba-b2d2-f1dab250e58d admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:46.697885 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b777dc-ed5e-4487-b9ba-085521b56458 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:46.698785 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b777dc-ed5e-4487-b9ba-085521b56458 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:46.699217 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b777dc-ed5e-4487-b9ba-085521b56458 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:46.699667 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b777dc-ed5e-4487-b9ba-085521b56458 admin admin] PATH_INFO: `/v3/projects/755c3e1da4fa473ba51d8ae01f8faa87` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:46.702982 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b777dc-ed5e-4487-b9ba-085521b56458 admin admin] RBAC: Authorizing `identity:delete_project(project_id=755c3e1da4fa473ba51d8ae01f8faa87)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:46.703876 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b777dc-ed5e-4487-b9ba-085521b56458 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:46.754785 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-46b777dc-ed5e-4487-b9ba-085521b56458 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '755c3e1da4fa473ba51d8ae01f8faa87', 'request_id': 'req-46b777dc-ed5e-4487-b9ba-085521b56458'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:46.761912 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-46b777dc-ed5e-4487-b9ba-085521b56458 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '755c3e1da4fa473ba51d8ae01f8faa87', 'request_id': 'req-46b777dc-ed5e-4487-b9ba-085521b56458'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:46.767652 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-46b777dc-ed5e-4487-b9ba-085521b56458 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '755c3e1da4fa473ba51d8ae01f8faa87', 'request_id': 'req-46b777dc-ed5e-4487-b9ba-085521b56458'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:46.779560 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-46b777dc-ed5e-4487-b9ba-085521b56458 admin admin] The token cache is being invalidate because project 755c3e1da4fa473ba51d8ae01f8faa87 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:25:46.779857 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-46b777dc-ed5e-4487-b9ba-085521b56458 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:46.780486 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-46b777dc-ed5e-4487-b9ba-085521b56458 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:46.781997 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 892/1806] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:25:46 2022] DELETE /identity/v3/projects/755c3e1da4fa473ba51d8ae01f8faa87 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:46.787848 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3d2a72-bc3f-4e15-86d9-cb6d586923ef None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:46.788301 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3d2a72-bc3f-4e15-86d9-cb6d586923ef None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:46.788414 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3d2a72-bc3f-4e15-86d9-cb6d586923ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:46.808569 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-4f3d2a72-bc3f-4e15-86d9-cb6d586923ef None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:46.809803 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-4f3d2a72-bc3f-4e15-86d9-cb6d586923ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:46.836937 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:46.877924 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 915/1807] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:25:46.885801 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b186e7e5-f182-4b52-8893-8be0299be2c3 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:46.886086 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b186e7e5-f182-4b52-8893-8be0299be2c3 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:46.886358 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b186e7e5-f182-4b52-8893-8be0299be2c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:46.906315 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-b186e7e5-f182-4b52-8893-8be0299be2c3 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:46.907532 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-b186e7e5-f182-4b52-8893-8be0299be2c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:46.916278 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 893/1808] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:25:46.920401 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3d2a72-bc3f-4e15-86d9-cb6d586923ef None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:46.931015 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3d2a72-bc3f-4e15-86d9-cb6d586923ef None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:46.935813 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0b8282-831f-4d5d-9153-89594b287d89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:46.936427 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0b8282-831f-4d5d-9153-89594b287d89 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:46.936703 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0b8282-831f-4d5d-9153-89594b287d89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:46.936968 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0b8282-831f-4d5d-9153-89594b287d89 admin admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:46.937638 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c0b8282-831f-4d5d-9153-89594b287d89 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:25:46.937932 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c0b8282-831f-4d5d-9153-89594b287d89 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:46.939499 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c0b8282-831f-4d5d-9153-89594b287d89 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:46.951133 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 916/1809] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:25:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:25:46.955522 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b186e7e5-f182-4b52-8893-8be0299be2c3 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:46.965344 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b186e7e5-f182-4b52-8893-8be0299be2c3 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:46.970005 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07262245-1421-44e0-a2e9-98a456527c5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:46.970590 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07262245-1421-44e0-a2e9-98a456527c5d admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:46.970859 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07262245-1421-44e0-a2e9-98a456527c5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:46.971134 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07262245-1421-44e0-a2e9-98a456527c5d admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:46.971794 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07262245-1421-44e0-a2e9-98a456527c5d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:46.974585 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07262245-1421-44e0-a2e9-98a456527c5d admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:47.000462 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 894/1810] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:25:46 2022] POST /identity/v3/projects => generated 371 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:25:47.005180 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0b8282-831f-4d5d-9153-89594b287d89 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:47.015043 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0b8282-831f-4d5d-9153-89594b287d89 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:47.071411 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf06fe72-a583-40a4-b6aa-ddcecda5869b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:47.072014 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf06fe72-a583-40a4-b6aa-ddcecda5869b admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:47.072292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf06fe72-a583-40a4-b6aa-ddcecda5869b admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:47.072561 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf06fe72-a583-40a4-b6aa-ddcecda5869b admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:47.073228 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf06fe72-a583-40a4-b6aa-ddcecda5869b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:47.075139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf06fe72-a583-40a4-b6aa-ddcecda5869b admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:47.096200 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 917/1811] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:25:47 2022] POST /identity/v3/users => generated 286 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:25:47.100438 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07262245-1421-44e0-a2e9-98a456527c5d admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:47.110761 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07262245-1421-44e0-a2e9-98a456527c5d admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:47.116109 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6599e3d-0c8a-4dd2-a25f-e8ad5bcb27ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:47.116705 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6599e3d-0c8a-4dd2-a25f-e8ad5bcb27ce admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:47.116979 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6599e3d-0c8a-4dd2-a25f-e8ad5bcb27ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:47.117283 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6599e3d-0c8a-4dd2-a25f-e8ad5bcb27ce admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:47.117917 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6599e3d-0c8a-4dd2-a25f-e8ad5bcb27ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:47.119494 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6599e3d-0c8a-4dd2-a25f-e8ad5bcb27ce admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:47.130047 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 895/1812] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:25:47 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:25:47.134088 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf06fe72-a583-40a4-b6aa-ddcecda5869b admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:47.143774 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf06fe72-a583-40a4-b6aa-ddcecda5869b admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:47.148918 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-820f30ba-fe4a-41a5-813b-4e4875bb6adc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:47.149602 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820f30ba-fe4a-41a5-813b-4e4875bb6adc admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:47.149879 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820f30ba-fe4a-41a5-813b-4e4875bb6adc admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:47.150144 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820f30ba-fe4a-41a5-813b-4e4875bb6adc admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:47.150771 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-820f30ba-fe4a-41a5-813b-4e4875bb6adc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:47.152337 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-820f30ba-fe4a-41a5-813b-4e4875bb6adc admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:47.162865 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 918/1813] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:25:47 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:25:47.166797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6599e3d-0c8a-4dd2-a25f-e8ad5bcb27ce admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:47.176679 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6599e3d-0c8a-4dd2-a25f-e8ad5bcb27ce admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:47.181479 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561f4413-b9ef-452d-96e7-702121fcf177 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:47.182114 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561f4413-b9ef-452d-96e7-702121fcf177 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:47.182419 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561f4413-b9ef-452d-96e7-702121fcf177 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:47.182708 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561f4413-b9ef-452d-96e7-702121fcf177 admin admin] PATH_INFO: `/v3/projects/a816bcfd87b34071a24b948c351811c4/users/1505f37c15c0461a912d59172ebdf94a/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:47.197667 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561f4413-b9ef-452d-96e7-702121fcf177 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a816bcfd87b34071a24b948c351811c4, user_id=1505f37c15c0461a912d59172ebdf94a, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:47.200319 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561f4413-b9ef-452d-96e7-702121fcf177 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:47.203305 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=85,keystone:DELETE=11,keystone:INSERT=4 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:47.224359 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 896/1814] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:25:47 2022] PUT /identity/v3/projects/a816bcfd87b34071a24b948c351811c4/users/1505f37c15c0461a912d59172ebdf94a/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:47.229396 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a8eb2c-347c-4e82-93d4-306451c32e15 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:47.229749 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a8eb2c-347c-4e82-93d4-306451c32e15 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:47.230128 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a8eb2c-347c-4e82-93d4-306451c32e15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:47.249085 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-33a8eb2c-347c-4e82-93d4-306451c32e15 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:47.250384 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-33a8eb2c-347c-4e82-93d4-306451c32e15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:47.314482 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 919/1815] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:25:47.324387 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561f4413-b9ef-452d-96e7-702121fcf177 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:47.326265 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-561f4413-b9ef-452d-96e7-702121fcf177 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:47.386614 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561f4413-b9ef-452d-96e7-702121fcf177 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:47.390194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f560f8-3101-49c9-88d9-bdec73b35b7c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:47.390804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f560f8-3101-49c9-88d9-bdec73b35b7c service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:47.391080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f560f8-3101-49c9-88d9-bdec73b35b7c service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:47.391412 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f560f8-3101-49c9-88d9-bdec73b35b7c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:47.401942 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f560f8-3101-49c9-88d9-bdec73b35b7c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:47.403411 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f560f8-3101-49c9-88d9-bdec73b35b7c service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:47.416410 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 897/1816] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:25:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:25:48.078988 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51915d51-877d-4408-88f6-8736f16664e4 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:48.079555 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51915d51-877d-4408-88f6-8736f16664e4 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:48.079555 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51915d51-877d-4408-88f6-8736f16664e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:48.112227 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-51915d51-877d-4408-88f6-8736f16664e4 None None] MFA Rules not processed for user `1505f37c15c0461a912d59172ebdf94a`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:48.113654 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-51915d51-877d-4408-88f6-8736f16664e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:48.142395 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:48.192709 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 920/1817] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:48 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:25:48.202883 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f560f8-3101-49c9-88d9-bdec73b35b7c service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:48.205332 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-52f560f8-3101-49c9-88d9-bdec73b35b7c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:48.263788 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f560f8-3101-49c9-88d9-bdec73b35b7c service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:48.267447 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da98e81d-233d-4843-8d94-5656ba0db22e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:48.268047 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da98e81d-233d-4843-8d94-5656ba0db22e service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:48.268326 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da98e81d-233d-4843-8d94-5656ba0db22e service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:48.268598 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da98e81d-233d-4843-8d94-5656ba0db22e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:48.276971 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da98e81d-233d-4843-8d94-5656ba0db22e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:48.278436 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da98e81d-233d-4843-8d94-5656ba0db22e service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:48.287210 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 898/1818] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:25:48 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:25:52.811117 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d45be94-02d2-4cc1-b13a-7a7ef3ef1907 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:52.811657 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d45be94-02d2-4cc1-b13a-7a7ef3ef1907 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:52.811657 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d45be94-02d2-4cc1-b13a-7a7ef3ef1907 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:52.830763 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-9d45be94-02d2-4cc1-b13a-7a7ef3ef1907 None None] MFA Rules not processed for user `1505f37c15c0461a912d59172ebdf94a`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:52.832443 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-9d45be94-02d2-4cc1-b13a-7a7ef3ef1907 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:52.839831 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 921/1819] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:52 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:25:54.600523 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da98e81d-233d-4843-8d94-5656ba0db22e service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:54.601985 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-da98e81d-233d-4843-8d94-5656ba0db22e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:54.618303 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da98e81d-233d-4843-8d94-5656ba0db22e service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:54.624949 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7f4618-abd0-443b-ad27-1c743464581d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:54.625758 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7f4618-abd0-443b-ad27-1c743464581d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:54.626098 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7f4618-abd0-443b-ad27-1c743464581d admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:54.626402 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7f4618-abd0-443b-ad27-1c743464581d admin admin] PATH_INFO: `/v3/users/d411dccf786346b1abfa3b2d3f9736b0` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:54.627570 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7f4618-abd0-443b-ad27-1c743464581d admin admin] RBAC: Authorizing `identity:delete_user(user_id=d411dccf786346b1abfa3b2d3f9736b0)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:54.630222 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7f4618-abd0-443b-ad27-1c743464581d admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:54.686295 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bb7f4618-abd0-443b-ad27-1c743464581d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd411dccf786346b1abfa3b2d3f9736b0'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:54.694427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bb7f4618-abd0-443b-ad27-1c743464581d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd411dccf786346b1abfa3b2d3f9736b0'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:54.695068 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bb7f4618-abd0-443b-ad27-1c743464581d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd411dccf786346b1abfa3b2d3f9736b0'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:54.709354 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bb7f4618-abd0-443b-ad27-1c743464581d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd411dccf786346b1abfa3b2d3f9736b0'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:54.716405 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bb7f4618-abd0-443b-ad27-1c743464581d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd411dccf786346b1abfa3b2d3f9736b0'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:54.723803 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 899/1820] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:25:54 2022] DELETE /identity/v3/users/d411dccf786346b1abfa3b2d3f9736b0 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:54.727624 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39a0cb0-17a0-4983-b379-c0a53f25708c None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:54.728102 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39a0cb0-17a0-4983-b379-c0a53f25708c None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:54.728299 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39a0cb0-17a0-4983-b379-c0a53f25708c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:54.749135 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-e39a0cb0-17a0-4983-b379-c0a53f25708c None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:54.750613 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-e39a0cb0-17a0-4983-b379-c0a53f25708c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:54.822478 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 922/1821] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:25:54.835984 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7f4618-abd0-443b-ad27-1c743464581d admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:54.837577 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-bb7f4618-abd0-443b-ad27-1c743464581d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:54.855216 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:54.934824 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7f4618-abd0-443b-ad27-1c743464581d admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:54.938585 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e2297c0-fdcc-48fc-aed5-462451904813 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:54.939276 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2297c0-fdcc-48fc-aed5-462451904813 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:54.939617 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2297c0-fdcc-48fc-aed5-462451904813 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:54.939982 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2297c0-fdcc-48fc-aed5-462451904813 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:54.950350 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e2297c0-fdcc-48fc-aed5-462451904813 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:54.951888 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e2297c0-fdcc-48fc-aed5-462451904813 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:54.962676 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 900/1822] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:25:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:25:55.386015 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39a0cb0-17a0-4983-b379-c0a53f25708c None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:55.387425 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-e39a0cb0-17a0-4983-b379-c0a53f25708c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:55.400556 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39a0cb0-17a0-4983-b379-c0a53f25708c None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:55.405499 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ea5383-7b90-4d62-a037-6260817ee538 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:55.406169 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ea5383-7b90-4d62-a037-6260817ee538 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:55.406447 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ea5383-7b90-4d62-a037-6260817ee538 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:55.406719 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ea5383-7b90-4d62-a037-6260817ee538 admin admin] PATH_INFO: `/v3/projects/218f9a606c4f493eb8eec0347f706043` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:55.407862 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ea5383-7b90-4d62-a037-6260817ee538 admin admin] RBAC: Authorizing `identity:delete_project(project_id=218f9a606c4f493eb8eec0347f706043)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:55.409835 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ea5383-7b90-4d62-a037-6260817ee538 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:55.462457 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-13ea5383-7b90-4d62-a037-6260817ee538 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '218f9a606c4f493eb8eec0347f706043', 'request_id': 'req-13ea5383-7b90-4d62-a037-6260817ee538'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:55.470510 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-13ea5383-7b90-4d62-a037-6260817ee538 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '218f9a606c4f493eb8eec0347f706043', 'request_id': 'req-13ea5383-7b90-4d62-a037-6260817ee538'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:55.477866 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-13ea5383-7b90-4d62-a037-6260817ee538 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '218f9a606c4f493eb8eec0347f706043', 'request_id': 'req-13ea5383-7b90-4d62-a037-6260817ee538'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:55.490188 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-13ea5383-7b90-4d62-a037-6260817ee538 admin admin] The token cache is being invalidate because project 218f9a606c4f493eb8eec0347f706043 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:25:55.490188 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-13ea5383-7b90-4d62-a037-6260817ee538 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:55.490682 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-13ea5383-7b90-4d62-a037-6260817ee538 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:55.492207 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 923/1823] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:25:55 2022] DELETE /identity/v3/projects/218f9a606c4f493eb8eec0347f706043 => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:55.497028 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e30e96d-97b8-4a72-b520-a15db6401c01 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:55.497337 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e30e96d-97b8-4a72-b520-a15db6401c01 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:55.497615 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e30e96d-97b8-4a72-b520-a15db6401c01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:55.523145 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-5e30e96d-97b8-4a72-b520-a15db6401c01 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:55.524887 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-5e30e96d-97b8-4a72-b520-a15db6401c01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:55.591482 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 901/1824] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:25:55.599547 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafc9347-fdeb-474c-a911-fc185ab05152 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:55.599894 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafc9347-fdeb-474c-a911-fc185ab05152 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:55.600277 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafc9347-fdeb-474c-a911-fc185ab05152 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:55.623414 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-cafc9347-fdeb-474c-a911-fc185ab05152 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:55.624891 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-cafc9347-fdeb-474c-a911-fc185ab05152 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:55.634323 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 924/1825] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:25:55.638827 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e30e96d-97b8-4a72-b520-a15db6401c01 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:55.650660 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e30e96d-97b8-4a72-b520-a15db6401c01 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:55.656138 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea646296-8c9d-48b7-be10-543787114e47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:55.657258 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea646296-8c9d-48b7-be10-543787114e47 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:55.657540 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea646296-8c9d-48b7-be10-543787114e47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:55.657807 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea646296-8c9d-48b7-be10-543787114e47 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:55.658468 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea646296-8c9d-48b7-be10-543787114e47 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:25:55.658767 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea646296-8c9d-48b7-be10-543787114e47 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:55.660403 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea646296-8c9d-48b7-be10-543787114e47 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:55.676140 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 902/1826] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:25:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:25:55.686350 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafc9347-fdeb-474c-a911-fc185ab05152 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:55.700017 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafc9347-fdeb-474c-a911-fc185ab05152 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:55.705532 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14dae027-518c-4f72-9890-ca1e5385f04b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:55.706394 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14dae027-518c-4f72-9890-ca1e5385f04b admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:55.706893 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14dae027-518c-4f72-9890-ca1e5385f04b admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:55.707905 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14dae027-518c-4f72-9890-ca1e5385f04b admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:55.708759 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14dae027-518c-4f72-9890-ca1e5385f04b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:55.711037 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14dae027-518c-4f72-9890-ca1e5385f04b admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:55.740838 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 925/1827] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:25:55 2022] POST /identity/v3/projects => generated 359 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:25:55.745081 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea646296-8c9d-48b7-be10-543787114e47 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:55.757744 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea646296-8c9d-48b7-be10-543787114e47 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:55.817822 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58cc639-b188-41f2-931a-4937a8374667 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:55.817822 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58cc639-b188-41f2-931a-4937a8374667 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:55.817822 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58cc639-b188-41f2-931a-4937a8374667 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:55.818090 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58cc639-b188-41f2-931a-4937a8374667 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:55.818653 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58cc639-b188-41f2-931a-4937a8374667 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:55.820570 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58cc639-b188-41f2-931a-4937a8374667 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:55.840773 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 903/1828] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:25:55 2022] POST /identity/v3/users => generated 280 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:25:55.844913 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14dae027-518c-4f72-9890-ca1e5385f04b admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:55.855137 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14dae027-518c-4f72-9890-ca1e5385f04b admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:55.859807 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02712f4d-bd5d-43ae-a771-636e2527a41f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:55.860463 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02712f4d-bd5d-43ae-a771-636e2527a41f admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:55.860739 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02712f4d-bd5d-43ae-a771-636e2527a41f admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:55.861005 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02712f4d-bd5d-43ae-a771-636e2527a41f admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:55.861681 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02712f4d-bd5d-43ae-a771-636e2527a41f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:55.863825 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02712f4d-bd5d-43ae-a771-636e2527a41f admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:55.873667 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 926/1829] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:25:55 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:25:55.877415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58cc639-b188-41f2-931a-4937a8374667 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:55.887927 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58cc639-b188-41f2-931a-4937a8374667 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:55.892557 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9506ab3-b0cf-4f01-b6e6-07e4043d5c8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:55.893289 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9506ab3-b0cf-4f01-b6e6-07e4043d5c8e admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:55.893566 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9506ab3-b0cf-4f01-b6e6-07e4043d5c8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:55.893836 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9506ab3-b0cf-4f01-b6e6-07e4043d5c8e admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:55.894540 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9506ab3-b0cf-4f01-b6e6-07e4043d5c8e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:55.896123 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9506ab3-b0cf-4f01-b6e6-07e4043d5c8e admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:55.911283 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 904/1830] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:25:55 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:25:55.915804 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02712f4d-bd5d-43ae-a771-636e2527a41f admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:55.923159 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:55.928484 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02712f4d-bd5d-43ae-a771-636e2527a41f admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:55.934775 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bead301e-6582-4b62-bc17-c66f24e74067 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:55.935600 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bead301e-6582-4b62-bc17-c66f24e74067 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:55.936053 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bead301e-6582-4b62-bc17-c66f24e74067 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:55.936371 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bead301e-6582-4b62-bc17-c66f24e74067 admin admin] PATH_INFO: `/v3/projects/b25b815f8e724639bd8aa8f6ca470c52/users/6e8f080cc46c4a01b9d6395b29cd2b90/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:55.953583 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bead301e-6582-4b62-bc17-c66f24e74067 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b25b815f8e724639bd8aa8f6ca470c52, user_id=6e8f080cc46c4a01b9d6395b29cd2b90, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:55.956310 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bead301e-6582-4b62-bc17-c66f24e74067 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:55.968881 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 927/1831] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:25:55 2022] PUT /identity/v3/projects/b25b815f8e724639bd8aa8f6ca470c52/users/6e8f080cc46c4a01b9d6395b29cd2b90/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:55.979458 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82beef7-d7fc-4cd6-97a3-055b94d2cd85 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:55.979744 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82beef7-d7fc-4cd6-97a3-055b94d2cd85 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:55.980016 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82beef7-d7fc-4cd6-97a3-055b94d2cd85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:56.008325 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-d82beef7-d7fc-4cd6-97a3-055b94d2cd85 None None] MFA Rules not processed for user `6e8f080cc46c4a01b9d6395b29cd2b90`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:56.009582 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d82beef7-d7fc-4cd6-97a3-055b94d2cd85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:56.018495 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:56.073251 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 905/1832] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:55 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:25:56.108008 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bead301e-6582-4b62-bc17-c66f24e74067 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:56.109727 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-bead301e-6582-4b62-bc17-c66f24e74067 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:56.168269 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bead301e-6582-4b62-bc17-c66f24e74067 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:56.175734 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873f8aba-9e44-4581-95fe-62ff048e1aea service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:56.176505 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873f8aba-9e44-4581-95fe-62ff048e1aea service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:56.176752 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873f8aba-9e44-4581-95fe-62ff048e1aea service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:56.177024 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873f8aba-9e44-4581-95fe-62ff048e1aea service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:56.188752 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-873f8aba-9e44-4581-95fe-62ff048e1aea service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:56.190221 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-873f8aba-9e44-4581-95fe-62ff048e1aea service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:56.201285 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 928/1833] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:25:56 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:25:56.218220 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82beef7-d7fc-4cd6-97a3-055b94d2cd85 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:56.285595 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82beef7-d7fc-4cd6-97a3-055b94d2cd85 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:56.290876 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa25e791-44b6-4eeb-be84-7a8f7fb64e4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:56.291508 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa25e791-44b6-4eeb-be84-7a8f7fb64e4d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:56.291784 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa25e791-44b6-4eeb-be84-7a8f7fb64e4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:56.292051 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa25e791-44b6-4eeb-be84-7a8f7fb64e4d admin admin] PATH_INFO: `/v3/users/6e8f080cc46c4a01b9d6395b29cd2b90` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:56.293232 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa25e791-44b6-4eeb-be84-7a8f7fb64e4d admin admin] RBAC: Authorizing `identity:delete_user(user_id=6e8f080cc46c4a01b9d6395b29cd2b90)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:56.295167 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa25e791-44b6-4eeb-be84-7a8f7fb64e4d admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:56.343637 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-fa25e791-44b6-4eeb-be84-7a8f7fb64e4d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6e8f080cc46c4a01b9d6395b29cd2b90'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:56.352983 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-fa25e791-44b6-4eeb-be84-7a8f7fb64e4d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6e8f080cc46c4a01b9d6395b29cd2b90'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:56.353719 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-fa25e791-44b6-4eeb-be84-7a8f7fb64e4d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6e8f080cc46c4a01b9d6395b29cd2b90'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:56.396029 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-fa25e791-44b6-4eeb-be84-7a8f7fb64e4d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6e8f080cc46c4a01b9d6395b29cd2b90'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:56.405339 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-fa25e791-44b6-4eeb-be84-7a8f7fb64e4d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6e8f080cc46c4a01b9d6395b29cd2b90'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:56.412668 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 906/1834] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:25:56 2022] DELETE /identity/v3/users/6e8f080cc46c4a01b9d6395b29cd2b90 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:56.417397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb259940-1a57-45a4-ae35-77d6fe3c51cd None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:56.417685 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb259940-1a57-45a4-ae35-77d6fe3c51cd None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:56.417962 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb259940-1a57-45a4-ae35-77d6fe3c51cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:56.440627 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-fb259940-1a57-45a4-ae35-77d6fe3c51cd None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:56.442339 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-fb259940-1a57-45a4-ae35-77d6fe3c51cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:56.504604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 929/1835] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:25:56.514375 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa25e791-44b6-4eeb-be84-7a8f7fb64e4d admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:56.515840 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-fa25e791-44b6-4eeb-be84-7a8f7fb64e4d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:56.554326 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:56.581249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa25e791-44b6-4eeb-be84-7a8f7fb64e4d admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:56.586642 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d98ced1-d7d7-450e-b503-361d122eb731 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:56.587251 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d98ced1-d7d7-450e-b503-361d122eb731 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:56.587527 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d98ced1-d7d7-450e-b503-361d122eb731 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:56.587797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d98ced1-d7d7-450e-b503-361d122eb731 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:56.598282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d98ced1-d7d7-450e-b503-361d122eb731 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:56.599856 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d98ced1-d7d7-450e-b503-361d122eb731 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:56.609729 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 907/1836] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:25:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:25:56.993330 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb259940-1a57-45a4-ae35-77d6fe3c51cd None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:56.994776 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-fb259940-1a57-45a4-ae35-77d6fe3c51cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:57.007280 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb259940-1a57-45a4-ae35-77d6fe3c51cd None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:57.011927 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05946ded-053d-4b93-9916-e017009303b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:57.012561 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05946ded-053d-4b93-9916-e017009303b3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:57.012840 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05946ded-053d-4b93-9916-e017009303b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:57.013147 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05946ded-053d-4b93-9916-e017009303b3 admin admin] PATH_INFO: `/v3/projects/b25b815f8e724639bd8aa8f6ca470c52` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:57.014263 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05946ded-053d-4b93-9916-e017009303b3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b25b815f8e724639bd8aa8f6ca470c52)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:57.016169 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05946ded-053d-4b93-9916-e017009303b3 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:57.064534 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-05946ded-053d-4b93-9916-e017009303b3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b25b815f8e724639bd8aa8f6ca470c52', 'request_id': 'req-05946ded-053d-4b93-9916-e017009303b3'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:57.084799 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-05946ded-053d-4b93-9916-e017009303b3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b25b815f8e724639bd8aa8f6ca470c52', 'request_id': 'req-05946ded-053d-4b93-9916-e017009303b3'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:57.091594 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-05946ded-053d-4b93-9916-e017009303b3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b25b815f8e724639bd8aa8f6ca470c52', 'request_id': 'req-05946ded-053d-4b93-9916-e017009303b3'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:57.102804 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-05946ded-053d-4b93-9916-e017009303b3 admin admin] The token cache is being invalidate because project b25b815f8e724639bd8aa8f6ca470c52 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:25:57.103093 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-05946ded-053d-4b93-9916-e017009303b3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:57.103725 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-05946ded-053d-4b93-9916-e017009303b3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:25:57.105251 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 930/1837] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:25:56 2022] DELETE /identity/v3/projects/b25b815f8e724639bd8aa8f6ca470c52 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:57.111674 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2a7573-3d33-48f6-a094-8eb23fc1e528 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:57.112126 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2a7573-3d33-48f6-a094-8eb23fc1e528 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:57.112240 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2a7573-3d33-48f6-a094-8eb23fc1e528 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:57.132385 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-ab2a7573-3d33-48f6-a094-8eb23fc1e528 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:57.133624 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ab2a7573-3d33-48f6-a094-8eb23fc1e528 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:57.193042 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 908/1838] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:25:57.201856 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005f873b-d982-44d3-8c9a-0d1f74ce30d0 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:57.202145 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005f873b-d982-44d3-8c9a-0d1f74ce30d0 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:57.202416 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005f873b-d982-44d3-8c9a-0d1f74ce30d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:57.216440 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:57.222727 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-005f873b-d982-44d3-8c9a-0d1f74ce30d0 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:57.224596 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-005f873b-d982-44d3-8c9a-0d1f74ce30d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:57.232409 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 931/1839] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:25:57.236565 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2a7573-3d33-48f6-a094-8eb23fc1e528 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:57.247248 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2a7573-3d33-48f6-a094-8eb23fc1e528 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:57.251765 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc36b51b-f4d5-4403-b016-935d538843d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:57.252351 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc36b51b-f4d5-4403-b016-935d538843d4 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:57.252625 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc36b51b-f4d5-4403-b016-935d538843d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:57.252896 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc36b51b-f4d5-4403-b016-935d538843d4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:57.254191 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc36b51b-f4d5-4403-b016-935d538843d4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:25:57.254889 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc36b51b-f4d5-4403-b016-935d538843d4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:57.256472 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc36b51b-f4d5-4403-b016-935d538843d4 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:57.267669 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 909/1840] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:25:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:25:57.271836 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-005f873b-d982-44d3-8c9a-0d1f74ce30d0 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:57.280904 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-005f873b-d982-44d3-8c9a-0d1f74ce30d0 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:57.285774 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744a39e4-5eee-4e7f-9a23-9d76a4c052f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:57.286363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744a39e4-5eee-4e7f-9a23-9d76a4c052f9 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:57.286639 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744a39e4-5eee-4e7f-9a23-9d76a4c052f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:57.286907 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744a39e4-5eee-4e7f-9a23-9d76a4c052f9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:57.287526 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744a39e4-5eee-4e7f-9a23-9d76a4c052f9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:57.289476 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744a39e4-5eee-4e7f-9a23-9d76a4c052f9 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:57.317275 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 932/1841] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:25:57 2022] POST /identity/v3/projects => generated 373 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:25:57.321380 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc36b51b-f4d5-4403-b016-935d538843d4 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:57.331505 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc36b51b-f4d5-4403-b016-935d538843d4 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:57.388211 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1286fe2d-dcc0-4850-9abe-3c5bb690aba1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:57.388815 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1286fe2d-dcc0-4850-9abe-3c5bb690aba1 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:57.389120 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1286fe2d-dcc0-4850-9abe-3c5bb690aba1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:57.389442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1286fe2d-dcc0-4850-9abe-3c5bb690aba1 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:57.391026 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1286fe2d-dcc0-4850-9abe-3c5bb690aba1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:57.391964 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1286fe2d-dcc0-4850-9abe-3c5bb690aba1 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:57.414298 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 910/1842] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:25:57 2022] POST /identity/v3/users => generated 287 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:25:57.418398 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744a39e4-5eee-4e7f-9a23-9d76a4c052f9 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:57.429385 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744a39e4-5eee-4e7f-9a23-9d76a4c052f9 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:57.434574 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d994b8-d951-4f64-9419-f3becf5943b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:57.435207 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d994b8-d951-4f64-9419-f3becf5943b2 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:57.435458 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d994b8-d951-4f64-9419-f3becf5943b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:57.436097 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d994b8-d951-4f64-9419-f3becf5943b2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:57.436623 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d994b8-d951-4f64-9419-f3becf5943b2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:57.438252 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d994b8-d951-4f64-9419-f3becf5943b2 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:57.452034 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 933/1843] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:25:57 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:25:57.453270 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1286fe2d-dcc0-4850-9abe-3c5bb690aba1 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:57.463508 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1286fe2d-dcc0-4850-9abe-3c5bb690aba1 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:57.468152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ad16f9-2ed4-43f2-a0ae-bf242e4b0988 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:57.468751 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ad16f9-2ed4-43f2-a0ae-bf242e4b0988 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:57.469029 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ad16f9-2ed4-43f2-a0ae-bf242e4b0988 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:57.469334 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ad16f9-2ed4-43f2-a0ae-bf242e4b0988 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:57.469972 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59ad16f9-2ed4-43f2-a0ae-bf242e4b0988 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:57.471565 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59ad16f9-2ed4-43f2-a0ae-bf242e4b0988 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:57.484324 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 911/1844] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:25:57 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:25:57.486655 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d994b8-d951-4f64-9419-f3becf5943b2 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:57.497320 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d994b8-d951-4f64-9419-f3becf5943b2 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:57.501939 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac50f8d-530c-4237-b09f-8e83629041c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:57.502873 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac50f8d-530c-4237-b09f-8e83629041c2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:57.503158 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac50f8d-530c-4237-b09f-8e83629041c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:57.504031 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac50f8d-530c-4237-b09f-8e83629041c2 admin admin] PATH_INFO: `/v3/projects/40ecdf4e71844f7f9e3da7f59659b605/users/181df03f4bba4f96afc3c888b3ca9b4d/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:57.521135 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac50f8d-530c-4237-b09f-8e83629041c2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=40ecdf4e71844f7f9e3da7f59659b605, user_id=181df03f4bba4f96afc3c888b3ca9b4d, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:57.524415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac50f8d-530c-4237-b09f-8e83629041c2 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:57.540229 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 934/1845] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:25:57 2022] PUT /identity/v3/projects/40ecdf4e71844f7f9e3da7f59659b605/users/181df03f4bba4f96afc3c888b3ca9b4d/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:25:57.545396 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbced49-fcfc-4bd0-ad5a-6a9967f1a490 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:57.545685 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbced49-fcfc-4bd0-ad5a-6a9967f1a490 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:57.545964 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbced49-fcfc-4bd0-ad5a-6a9967f1a490 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:57.582298 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-3dbced49-fcfc-4bd0-ad5a-6a9967f1a490 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:25:57.583531 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-3dbced49-fcfc-4bd0-ad5a-6a9967f1a490 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:57.620901 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:25:57.654337 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 912/1846] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:25:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:25:57.664443 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac50f8d-530c-4237-b09f-8e83629041c2 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:25:57.666117 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-6ac50f8d-530c-4237-b09f-8e83629041c2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:25:57.721584 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac50f8d-530c-4237-b09f-8e83629041c2 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:25:57.726245 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dbd918b-c4e4-44c2-8a2c-afc0f6a7b244 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:25:57.726881 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dbd918b-c4e4-44c2-8a2c-afc0f6a7b244 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:25:57.727157 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dbd918b-c4e4-44c2-8a2c-afc0f6a7b244 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:25:57.727428 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dbd918b-c4e4-44c2-8a2c-afc0f6a7b244 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:25:57.736633 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dbd918b-c4e4-44c2-8a2c-afc0f6a7b244 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:25:57.738750 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dbd918b-c4e4-44c2-8a2c-afc0f6a7b244 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:25:57.747889 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 935/1847] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:25:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:26:02.656048 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ad1473-c398-4bf5-b5ac-6807d9446638 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:02.656581 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ad1473-c398-4bf5-b5ac-6807d9446638 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:02.656581 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ad1473-c398-4bf5-b5ac-6807d9446638 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:02.692836 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-92ad1473-c398-4bf5-b5ac-6807d9446638 None None] MFA Rules not processed for user `181df03f4bba4f96afc3c888b3ca9b4d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:02.694396 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-92ad1473-c398-4bf5-b5ac-6807d9446638 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:02.881187 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 913/1848] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:02 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:02.882961 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dbd918b-c4e4-44c2-8a2c-afc0f6a7b244 service neutron] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:02.904604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dbd918b-c4e4-44c2-8a2c-afc0f6a7b244 service neutron] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:02.917358 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68de759a-a328-41dd-8778-261a24bb39b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:02.917992 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68de759a-a328-41dd-8778-261a24bb39b1 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:02.918268 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68de759a-a328-41dd-8778-261a24bb39b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:02.918544 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68de759a-a328-41dd-8778-261a24bb39b1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:02.919919 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68de759a-a328-41dd-8778-261a24bb39b1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:02.922277 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68de759a-a328-41dd-8778-261a24bb39b1 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:02.973114 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 936/1849] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:26:02 2022] POST /identity/v3/projects => generated 371 bytes in 88 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:26:02.974928 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ad1473-c398-4bf5-b5ac-6807d9446638 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:02.987150 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ad1473-c398-4bf5-b5ac-6807d9446638 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:03.105235 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66103007-cf46-479f-bed6-4a35ef363fc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:03.105941 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66103007-cf46-479f-bed6-4a35ef363fc9 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:03.106392 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66103007-cf46-479f-bed6-4a35ef363fc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:03.106732 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66103007-cf46-479f-bed6-4a35ef363fc9 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:03.107451 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66103007-cf46-479f-bed6-4a35ef363fc9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:03.109485 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66103007-cf46-479f-bed6-4a35ef363fc9 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:03.140047 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 914/1850] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:26:02 2022] POST /identity/v3/users => generated 292 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:26:03.144503 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68de759a-a328-41dd-8778-261a24bb39b1 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:03.155824 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68de759a-a328-41dd-8778-261a24bb39b1 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:03.160879 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81bd6012-2c51-4524-95ad-f9b86a783b17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:03.161523 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81bd6012-2c51-4524-95ad-f9b86a783b17 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:03.161800 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81bd6012-2c51-4524-95ad-f9b86a783b17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:03.162071 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81bd6012-2c51-4524-95ad-f9b86a783b17 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:03.162710 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81bd6012-2c51-4524-95ad-f9b86a783b17 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:03.164278 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81bd6012-2c51-4524-95ad-f9b86a783b17 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:03.179619 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 937/1851] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:26:03 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:26:03.184029 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66103007-cf46-479f-bed6-4a35ef363fc9 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:03.195421 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66103007-cf46-479f-bed6-4a35ef363fc9 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:03.200701 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34291f7-3529-4548-b6e8-6a0191c3879f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:03.201432 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34291f7-3529-4548-b6e8-6a0191c3879f admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:03.201713 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34291f7-3529-4548-b6e8-6a0191c3879f admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:03.201988 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34291f7-3529-4548-b6e8-6a0191c3879f admin admin] PATH_INFO: `/v3/projects/6de71544137646399172a5df0043ebfd/users/9f0842753cea48fe9508de4fc4d15b5f/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:03.218112 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f34291f7-3529-4548-b6e8-6a0191c3879f admin admin] RBAC: Authorizing `identity:create_grant(project_id=6de71544137646399172a5df0043ebfd, user_id=9f0842753cea48fe9508de4fc4d15b5f, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:03.220893 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f34291f7-3529-4548-b6e8-6a0191c3879f admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:03.241228 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 915/1852] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:26:03 2022] PUT /identity/v3/projects/6de71544137646399172a5df0043ebfd/users/9f0842753cea48fe9508de4fc4d15b5f/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:03.242120 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81bd6012-2c51-4524-95ad-f9b86a783b17 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:03.258070 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81bd6012-2c51-4524-95ad-f9b86a783b17 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:03.324561 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa5fbef-afe9-4d08-b3bf-232f3ba86521 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:03.325222 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa5fbef-afe9-4d08-b3bf-232f3ba86521 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:03.325558 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa5fbef-afe9-4d08-b3bf-232f3ba86521 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:03.325761 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa5fbef-afe9-4d08-b3bf-232f3ba86521 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:03.326545 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa5fbef-afe9-4d08-b3bf-232f3ba86521 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:03.328143 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa5fbef-afe9-4d08-b3bf-232f3ba86521 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:03.348454 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 938/1853] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:26:03 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:26:03.352338 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34291f7-3529-4548-b6e8-6a0191c3879f admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:03.364594 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34291f7-3529-4548-b6e8-6a0191c3879f admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:03.369238 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7ad7f7-025e-42e6-a09f-4571045ea21f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:03.369910 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7ad7f7-025e-42e6-a09f-4571045ea21f admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:03.370184 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7ad7f7-025e-42e6-a09f-4571045ea21f admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:03.370453 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7ad7f7-025e-42e6-a09f-4571045ea21f admin admin] PATH_INFO: `/v3/projects/6de71544137646399172a5df0043ebfd/users/9f0842753cea48fe9508de4fc4d15b5f/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:03.372417 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be7ad7f7-025e-42e6-a09f-4571045ea21f admin admin] RBAC: Authorizing `identity:create_grant(project_id=6de71544137646399172a5df0043ebfd, user_id=9f0842753cea48fe9508de4fc4d15b5f, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:03.375211 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be7ad7f7-025e-42e6-a09f-4571045ea21f admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:03.391211 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 916/1854] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:26:03 2022] PUT /identity/v3/projects/6de71544137646399172a5df0043ebfd/users/9f0842753cea48fe9508de4fc4d15b5f/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:05.584810 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db78fa66-562b-4aa5-b722-818b29f4bfd1 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:05.585935 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db78fa66-562b-4aa5-b722-818b29f4bfd1 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:05.586677 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db78fa66-562b-4aa5-b722-818b29f4bfd1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:05.609457 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-db78fa66-562b-4aa5-b722-818b29f4bfd1 None None] MFA Rules not processed for user `7a3f8b5eb09f4ba29f4bd343bd91f6a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:05.611040 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-db78fa66-562b-4aa5-b722-818b29f4bfd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:05.617280 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:05.717330 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 939/1855] 10.222.0.42 () {58 vars in 1009 bytes} [Thu Oct 27 01:26:05 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:05.768620 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7ad7f7-025e-42e6-a09f-4571045ea21f admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:05.771712 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-be7ad7f7-025e-42e6-a09f-4571045ea21f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:05.844421 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:05.908660 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7ad7f7-025e-42e6-a09f-4571045ea21f admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:05.915361 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aeccb2b-48d6-436b-b9c3-d59c4e2b3f9a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77f2cf9e0c3541f9bccfb7e213d88609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:05.916332 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aeccb2b-48d6-436b-b9c3-d59c4e2b3f9a service swift] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:05.916632 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aeccb2b-48d6-436b-b9c3-d59c4e2b3f9a service swift] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:05.917929 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aeccb2b-48d6-436b-b9c3-d59c4e2b3f9a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:05.927667 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aeccb2b-48d6-436b-b9c3-d59c4e2b3f9a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:05.929559 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aeccb2b-48d6-436b-b9c3-d59c4e2b3f9a service swift] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:05.941656 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 917/1856] 10.222.0.42 () {60 vars in 1384 bytes} [Thu Oct 27 01:26:05 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 27 01:26:06.495662 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db78fa66-562b-4aa5-b722-818b29f4bfd1 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:06.497198 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-db78fa66-562b-4aa5-b722-818b29f4bfd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:06.635089 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db78fa66-562b-4aa5-b722-818b29f4bfd1 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:06.644743 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74222e3c-9765-4cbe-90dd-0318d183a88f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:06.646434 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74222e3c-9765-4cbe-90dd-0318d183a88f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:06.647061 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74222e3c-9765-4cbe-90dd-0318d183a88f admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:06.647899 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74222e3c-9765-4cbe-90dd-0318d183a88f admin admin] PATH_INFO: `/v3/users/e1c00aa46dac42f4a330cb3a6e871735` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:06.651475 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74222e3c-9765-4cbe-90dd-0318d183a88f admin admin] RBAC: Authorizing `identity:delete_user(user_id=e1c00aa46dac42f4a330cb3a6e871735)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:06.654536 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74222e3c-9765-4cbe-90dd-0318d183a88f admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:06.790115 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-74222e3c-9765-4cbe-90dd-0318d183a88f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e1c00aa46dac42f4a330cb3a6e871735'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:06.812126 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-74222e3c-9765-4cbe-90dd-0318d183a88f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e1c00aa46dac42f4a330cb3a6e871735'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:06.846736 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-74222e3c-9765-4cbe-90dd-0318d183a88f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e1c00aa46dac42f4a330cb3a6e871735'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:06.846736 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-74222e3c-9765-4cbe-90dd-0318d183a88f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e1c00aa46dac42f4a330cb3a6e871735'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:06.846736 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-74222e3c-9765-4cbe-90dd-0318d183a88f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e1c00aa46dac42f4a330cb3a6e871735'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:06.869169 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 940/1857] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:26:06 2022] DELETE /identity/v3/users/e1c00aa46dac42f4a330cb3a6e871735 => generated 0 bytes in 374 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:06.873660 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603d2083-61d8-4501-abac-be2e6efe2fb6 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:06.874140 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603d2083-61d8-4501-abac-be2e6efe2fb6 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:06.874738 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603d2083-61d8-4501-abac-be2e6efe2fb6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:06.917636 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-603d2083-61d8-4501-abac-be2e6efe2fb6 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:06.918890 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-603d2083-61d8-4501-abac-be2e6efe2fb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:07.090933 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 918/1858] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:26:07.139659 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248da9e2-2bbf-4650-8e41-a1d3537092d8 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:07.139943 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248da9e2-2bbf-4650-8e41-a1d3537092d8 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:07.140379 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248da9e2-2bbf-4650-8e41-a1d3537092d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:07.162960 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-603d2083-61d8-4501-abac-be2e6efe2fb6 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:07.167155 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-603d2083-61d8-4501-abac-be2e6efe2fb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:07.189352 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-248da9e2-2bbf-4650-8e41-a1d3537092d8 None None] MFA Rules not processed for user `7a3f8b5eb09f4ba29f4bd343bd91f6a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:07.191492 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-248da9e2-2bbf-4650-8e41-a1d3537092d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:07.260960 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:07.281384 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-603d2083-61d8-4501-abac-be2e6efe2fb6 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:07.287592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb0cefa-0766-489c-b5d7-0f8a8deefb9b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:07.288715 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb0cefa-0766-489c-b5d7-0f8a8deefb9b service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:07.289341 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb0cefa-0766-489c-b5d7-0f8a8deefb9b service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:07.289781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb0cefa-0766-489c-b5d7-0f8a8deefb9b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:07.303628 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb0cefa-0766-489c-b5d7-0f8a8deefb9b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:07.307067 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb0cefa-0766-489c-b5d7-0f8a8deefb9b service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:07.319315 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 941/1859] 10.222.0.42 () {58 vars in 967 bytes} [Thu Oct 27 01:26:07 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:07.328914 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 919/1860] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:26:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:26:07.332905 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248da9e2-2bbf-4650-8e41-a1d3537092d8 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:07.334567 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-248da9e2-2bbf-4650-8e41-a1d3537092d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:07.461990 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248da9e2-2bbf-4650-8e41-a1d3537092d8 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:07.478654 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7add27-b4d2-4a5e-9aa8-1ee736102d5d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77f2cf9e0c3541f9bccfb7e213d88609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:07.479604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7add27-b4d2-4a5e-9aa8-1ee736102d5d service swift] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:07.480091 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7add27-b4d2-4a5e-9aa8-1ee736102d5d service swift] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:07.480573 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7add27-b4d2-4a5e-9aa8-1ee736102d5d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:07.493330 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7add27-b4d2-4a5e-9aa8-1ee736102d5d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:07.495845 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7add27-b4d2-4a5e-9aa8-1ee736102d5d service swift] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:07.507688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 942/1861] 10.222.0.42 () {60 vars in 1384 bytes} [Thu Oct 27 01:26:07 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 175 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 27 01:26:07.995791 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12db72c2-fce9-4abb-a2b9-e59f0b51fc74 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:07.996317 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12db72c2-fce9-4abb-a2b9-e59f0b51fc74 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:07.996990 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12db72c2-fce9-4abb-a2b9-e59f0b51fc74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:08.047100 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-12db72c2-fce9-4abb-a2b9-e59f0b51fc74 None None] MFA Rules not processed for user `7a3f8b5eb09f4ba29f4bd343bd91f6a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:08.048586 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-12db72c2-fce9-4abb-a2b9-e59f0b51fc74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:08.070459 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 920/1862] 10.222.0.42 () {58 vars in 1009 bytes} [Thu Oct 27 01:26:07 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:08.082506 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7add27-b4d2-4a5e-9aa8-1ee736102d5d service swift] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:08.121908 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7add27-b4d2-4a5e-9aa8-1ee736102d5d service swift] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:08.137670 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0167c9d-fabe-43e6-8ed6-743022610c27 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77f2cf9e0c3541f9bccfb7e213d88609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:08.149012 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0167c9d-fabe-43e6-8ed6-743022610c27 service swift] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:08.150907 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0167c9d-fabe-43e6-8ed6-743022610c27 service swift] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:08.151365 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0167c9d-fabe-43e6-8ed6-743022610c27 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:08.178718 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0167c9d-fabe-43e6-8ed6-743022610c27 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:08.181104 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0167c9d-fabe-43e6-8ed6-743022610c27 service swift] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:08.201806 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 943/1863] 10.222.0.42 () {60 vars in 1384 bytes} [Thu Oct 27 01:26:08 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 27 01:26:08.469505 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12db72c2-fce9-4abb-a2b9-e59f0b51fc74 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:08.477022 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-12db72c2-fce9-4abb-a2b9-e59f0b51fc74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:08.499768 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12db72c2-fce9-4abb-a2b9-e59f0b51fc74 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:08.528841 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565766ca-5b73-4d49-af4b-e2344cfeb166 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:08.529539 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565766ca-5b73-4d49-af4b-e2344cfeb166 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:08.529819 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565766ca-5b73-4d49-af4b-e2344cfeb166 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:08.530091 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565766ca-5b73-4d49-af4b-e2344cfeb166 admin admin] PATH_INFO: `/v3/projects/452958c488164ad1ad36e13b7387cbc1` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:08.531323 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-565766ca-5b73-4d49-af4b-e2344cfeb166 admin admin] RBAC: Authorizing `identity:delete_project(project_id=452958c488164ad1ad36e13b7387cbc1)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:08.533283 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-565766ca-5b73-4d49-af4b-e2344cfeb166 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:08.657445 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:08.668221 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-565766ca-5b73-4d49-af4b-e2344cfeb166 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '452958c488164ad1ad36e13b7387cbc1', 'request_id': 'req-565766ca-5b73-4d49-af4b-e2344cfeb166'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:08.682636 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-565766ca-5b73-4d49-af4b-e2344cfeb166 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '452958c488164ad1ad36e13b7387cbc1', 'request_id': 'req-565766ca-5b73-4d49-af4b-e2344cfeb166'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:08.692154 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-565766ca-5b73-4d49-af4b-e2344cfeb166 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '452958c488164ad1ad36e13b7387cbc1', 'request_id': 'req-565766ca-5b73-4d49-af4b-e2344cfeb166'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:08.706931 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-565766ca-5b73-4d49-af4b-e2344cfeb166 admin admin] The token cache is being invalidate because project 452958c488164ad1ad36e13b7387cbc1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:26:08.707581 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-565766ca-5b73-4d49-af4b-e2344cfeb166 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:08.710419 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-565766ca-5b73-4d49-af4b-e2344cfeb166 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:08.720330 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 921/1864] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:26:08 2022] DELETE /identity/v3/projects/452958c488164ad1ad36e13b7387cbc1 => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:08.725038 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7013f5-04dd-4892-b47d-cf41786a9584 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:08.725297 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7013f5-04dd-4892-b47d-cf41786a9584 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:08.725571 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7013f5-04dd-4892-b47d-cf41786a9584 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:08.799827 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-6e7013f5-04dd-4892-b47d-cf41786a9584 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:08.802756 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-6e7013f5-04dd-4892-b47d-cf41786a9584 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:08.892242 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5840509-f3c4-4dd4-8b5f-3fdfe8b0d610 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:08.892764 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5840509-f3c4-4dd4-8b5f-3fdfe8b0d610 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:08.894240 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5840509-f3c4-4dd4-8b5f-3fdfe8b0d610 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:08.935860 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-a5840509-f3c4-4dd4-8b5f-3fdfe8b0d610 None None] MFA Rules not processed for user `7a3f8b5eb09f4ba29f4bd343bd91f6a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:08.937423 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-a5840509-f3c4-4dd4-8b5f-3fdfe8b0d610 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:09.007151 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 944/1865] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 277 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:09.009400 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298df6a3-9c36-41a8-96a3-f728da92913d None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:09.009761 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298df6a3-9c36-41a8-96a3-f728da92913d None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:09.010107 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298df6a3-9c36-41a8-96a3-f728da92913d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:09.055114 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-298df6a3-9c36-41a8-96a3-f728da92913d None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:09.056568 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-298df6a3-9c36-41a8-96a3-f728da92913d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:09.078739 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 922/1866] 10.222.0.42 () {58 vars in 967 bytes} [Thu Oct 27 01:26:08 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:09.089711 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5840509-f3c4-4dd4-8b5f-3fdfe8b0d610 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:09.091952 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-a5840509-f3c4-4dd4-8b5f-3fdfe8b0d610 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:09.098575 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 945/1867] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:26:09.103070 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298df6a3-9c36-41a8-96a3-f728da92913d None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:09.119988 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298df6a3-9c36-41a8-96a3-f728da92913d None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:09.138216 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7bd9c80-76bd-41de-959e-3edfa1f6de05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:09.138946 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bd9c80-76bd-41de-959e-3edfa1f6de05 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:09.139302 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bd9c80-76bd-41de-959e-3edfa1f6de05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:09.143981 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bd9c80-76bd-41de-959e-3edfa1f6de05 admin admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:09.144859 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7bd9c80-76bd-41de-959e-3edfa1f6de05 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:26:09.145419 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7bd9c80-76bd-41de-959e-3edfa1f6de05 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:09.147618 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7bd9c80-76bd-41de-959e-3edfa1f6de05 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:09.181112 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 946/1868] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:26:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:26:09.185176 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7bd9c80-76bd-41de-959e-3edfa1f6de05 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:09.205821 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7bd9c80-76bd-41de-959e-3edfa1f6de05 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:09.218062 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435dd6a4-de96-41af-9331-3745e066ad5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:09.221586 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435dd6a4-de96-41af-9331-3745e066ad5f admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:09.222586 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435dd6a4-de96-41af-9331-3745e066ad5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:09.223514 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435dd6a4-de96-41af-9331-3745e066ad5f admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:09.225400 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-435dd6a4-de96-41af-9331-3745e066ad5f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:09.233740 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-435dd6a4-de96-41af-9331-3745e066ad5f admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:09.257339 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5840509-f3c4-4dd4-8b5f-3fdfe8b0d610 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:09.267871 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b59d1f-cb00-4e40-bf00-563d8c71ce38 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77f2cf9e0c3541f9bccfb7e213d88609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:09.269569 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b59d1f-cb00-4e40-bf00-563d8c71ce38 service swift] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:09.272490 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b59d1f-cb00-4e40-bf00-563d8c71ce38 service swift] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:09.273081 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b59d1f-cb00-4e40-bf00-563d8c71ce38 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:09.289170 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 947/1869] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:26:09 2022] POST /identity/v3/projects => generated 377 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:26:09.290346 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435dd6a4-de96-41af-9331-3745e066ad5f admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:09.309355 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b59d1f-cb00-4e40-bf00-563d8c71ce38 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:09.313724 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b59d1f-cb00-4e40-bf00-563d8c71ce38 service swift] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:09.316983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:09.342782 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 923/1870] 10.222.0.42 () {60 vars in 1384 bytes} [Thu Oct 27 01:26:09 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 253 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 27 01:26:09.454867 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435dd6a4-de96-41af-9331-3745e066ad5f admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:09.482856 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb226c6-0f5e-443d-8395-e58ea119ef8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:09.483525 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb226c6-0f5e-443d-8395-e58ea119ef8d admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:09.483806 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb226c6-0f5e-443d-8395-e58ea119ef8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:09.484078 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb226c6-0f5e-443d-8395-e58ea119ef8d admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:09.484737 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb226c6-0f5e-443d-8395-e58ea119ef8d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:09.489494 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb226c6-0f5e-443d-8395-e58ea119ef8d admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:09.538132 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 948/1871] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:26:09 2022] POST /identity/v3/users => generated 289 bytes in 248 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:26:09.542027 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b59d1f-cb00-4e40-bf00-563d8c71ce38 service swift] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:09.573305 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b59d1f-cb00-4e40-bf00-563d8c71ce38 service swift] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:09.584712 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c598162a-7e6c-49b6-b51b-3f074dc2de76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:09.586249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c598162a-7e6c-49b6-b51b-3f074dc2de76 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:09.586535 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c598162a-7e6c-49b6-b51b-3f074dc2de76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:09.586809 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c598162a-7e6c-49b6-b51b-3f074dc2de76 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:09.587473 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c598162a-7e6c-49b6-b51b-3f074dc2de76 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:09.594941 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c598162a-7e6c-49b6-b51b-3f074dc2de76 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:09.611570 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 924/1872] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:26:09 2022] GET /identity/v3/roles => generated 1420 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 27 01:26:09.615912 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb226c6-0f5e-443d-8395-e58ea119ef8d admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:09.638563 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb226c6-0f5e-443d-8395-e58ea119ef8d admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:09.664569 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44c4c64-9de6-41d9-b373-5991ee10f5fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:09.665230 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44c4c64-9de6-41d9-b373-5991ee10f5fb admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:09.665511 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44c4c64-9de6-41d9-b373-5991ee10f5fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:09.665784 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44c4c64-9de6-41d9-b373-5991ee10f5fb admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:09.666439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44c4c64-9de6-41d9-b373-5991ee10f5fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:09.668082 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44c4c64-9de6-41d9-b373-5991ee10f5fb admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:09.690478 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 949/1873] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:26:09 2022] GET /identity/v3/roles => generated 1420 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:26:09.697146 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c598162a-7e6c-49b6-b51b-3f074dc2de76 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:09.712916 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c598162a-7e6c-49b6-b51b-3f074dc2de76 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:09.732933 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86f1e903-71aa-4a07-bf55-126bf008d607 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:09.734080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f1e903-71aa-4a07-bf55-126bf008d607 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:09.737523 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f1e903-71aa-4a07-bf55-126bf008d607 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:09.747998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f1e903-71aa-4a07-bf55-126bf008d607 admin admin] PATH_INFO: `/v3/projects/2a64b8cf904d450f9b85cbb97590d98e/users/3768beee62fe403489672b08c0113239/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:09.792374 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86f1e903-71aa-4a07-bf55-126bf008d607 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2a64b8cf904d450f9b85cbb97590d98e, user_id=3768beee62fe403489672b08c0113239, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:09.795777 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86f1e903-71aa-4a07-bf55-126bf008d607 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:09.842174 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 925/1874] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:26:09 2022] PUT /identity/v3/projects/2a64b8cf904d450f9b85cbb97590d98e/users/3768beee62fe403489672b08c0113239/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:09.852853 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6921e7cb-5444-4329-a81d-5adf43ef57be None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:09.853187 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6921e7cb-5444-4329-a81d-5adf43ef57be None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:09.853625 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6921e7cb-5444-4329-a81d-5adf43ef57be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:09.918273 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-6921e7cb-5444-4329-a81d-5adf43ef57be None None] MFA Rules not processed for user `3768beee62fe403489672b08c0113239`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:09.924679 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-6921e7cb-5444-4329-a81d-5adf43ef57be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:10.092457 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 950/1875] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:09 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 240 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:10.093469 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86f1e903-71aa-4a07-bf55-126bf008d607 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:10.134217 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=94,keystone:DELETE=4,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:10.173548 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86f1e903-71aa-4a07-bf55-126bf008d607 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:10.182709 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c8d129c-0ff0-496e-ba84-246ea3a9d251 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:10.184254 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8d129c-0ff0-496e-ba84-246ea3a9d251 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:10.184716 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8d129c-0ff0-496e-ba84-246ea3a9d251 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:10.185899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8d129c-0ff0-496e-ba84-246ea3a9d251 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:10.187105 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c8d129c-0ff0-496e-ba84-246ea3a9d251 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:10.189564 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c8d129c-0ff0-496e-ba84-246ea3a9d251 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:10.236060 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 926/1876] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:26:10 2022] POST /identity/v3/projects => generated 377 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:26:10.241073 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6921e7cb-5444-4329-a81d-5adf43ef57be None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:10.276231 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6921e7cb-5444-4329-a81d-5adf43ef57be None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:10.383117 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb149aab-198b-4e4b-b7b7-64aaa0bbbfd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:10.383802 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb149aab-198b-4e4b-b7b7-64aaa0bbbfd5 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:10.384080 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb149aab-198b-4e4b-b7b7-64aaa0bbbfd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:10.384350 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb149aab-198b-4e4b-b7b7-64aaa0bbbfd5 admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:10.384989 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb149aab-198b-4e4b-b7b7-64aaa0bbbfd5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:10.387369 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb149aab-198b-4e4b-b7b7-64aaa0bbbfd5 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:10.420274 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 951/1877] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:26:10 2022] POST /identity/v3/users => generated 289 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:26:10.421579 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c8d129c-0ff0-496e-ba84-246ea3a9d251 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:10.438605 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c8d129c-0ff0-496e-ba84-246ea3a9d251 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:10.446957 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96a9840-d263-4f3f-ade9-d9b9d40b1453 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:10.447721 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96a9840-d263-4f3f-ade9-d9b9d40b1453 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:10.447996 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96a9840-d263-4f3f-ade9-d9b9d40b1453 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:10.448535 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96a9840-d263-4f3f-ade9-d9b9d40b1453 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:10.449489 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e96a9840-d263-4f3f-ade9-d9b9d40b1453 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:10.451529 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e96a9840-d263-4f3f-ade9-d9b9d40b1453 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:10.471763 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 927/1878] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:26:10 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:26:10.473136 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb149aab-198b-4e4b-b7b7-64aaa0bbbfd5 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:10.484772 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb149aab-198b-4e4b-b7b7-64aaa0bbbfd5 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:10.498902 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35a3d2d-fb91-4369-bf76-d52047e32645 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:10.499594 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35a3d2d-fb91-4369-bf76-d52047e32645 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:10.499776 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35a3d2d-fb91-4369-bf76-d52047e32645 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:10.500044 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35a3d2d-fb91-4369-bf76-d52047e32645 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:10.500685 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d35a3d2d-fb91-4369-bf76-d52047e32645 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:10.502316 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d35a3d2d-fb91-4369-bf76-d52047e32645 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:10.511430 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:10.525768 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96a9840-d263-4f3f-ade9-d9b9d40b1453 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:10.526720 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 952/1879] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:26:10 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 27 01:26:10.539228 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96a9840-d263-4f3f-ade9-d9b9d40b1453 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:10.552065 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6877422-f1d4-4f79-957b-3ac24f166259 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:10.552936 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6877422-f1d4-4f79-957b-3ac24f166259 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:10.553428 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6877422-f1d4-4f79-957b-3ac24f166259 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:10.553688 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6877422-f1d4-4f79-957b-3ac24f166259 admin admin] PATH_INFO: `/v3/projects/f1a27696da84492e892d10e4bac4df38/users/596b58e918664148b8d97e9b94902bab/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:10.593606 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6877422-f1d4-4f79-957b-3ac24f166259 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f1a27696da84492e892d10e4bac4df38, user_id=596b58e918664148b8d97e9b94902bab, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:10.597935 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6877422-f1d4-4f79-957b-3ac24f166259 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:10.623634 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 928/1880] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:26:10 2022] PUT /identity/v3/projects/f1a27696da84492e892d10e4bac4df38/users/596b58e918664148b8d97e9b94902bab/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:10.640963 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394bdde6-91e4-4a24-8048-12a92d4cbab4 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:10.641264 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394bdde6-91e4-4a24-8048-12a92d4cbab4 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:10.641623 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394bdde6-91e4-4a24-8048-12a92d4cbab4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:10.707847 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-394bdde6-91e4-4a24-8048-12a92d4cbab4 None None] MFA Rules not processed for user `596b58e918664148b8d97e9b94902bab`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:10.710877 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-394bdde6-91e4-4a24-8048-12a92d4cbab4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:10.830575 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 953/1881] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:10 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:10.835385 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6877422-f1d4-4f79-957b-3ac24f166259 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:10.932053 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6877422-f1d4-4f79-957b-3ac24f166259 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:10.941587 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560b286e-41f7-4236-a07a-ec4e1771b51b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:10.943483 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560b286e-41f7-4236-a07a-ec4e1771b51b admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:10.943998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560b286e-41f7-4236-a07a-ec4e1771b51b admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:10.945197 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560b286e-41f7-4236-a07a-ec4e1771b51b admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:10.946236 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560b286e-41f7-4236-a07a-ec4e1771b51b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:10.948962 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560b286e-41f7-4236-a07a-ec4e1771b51b admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:11.019493 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-394bdde6-91e4-4a24-8048-12a92d4cbab4 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:11.019690 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 929/1882] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:26:10 2022] POST /identity/v3/projects => generated 375 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:26:11.057098 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-394bdde6-91e4-4a24-8048-12a92d4cbab4 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:11.243935 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd3cdaa-34b6-4684-87fd-d9eeb10e4905 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:11.245426 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd3cdaa-34b6-4684-87fd-d9eeb10e4905 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:11.245426 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd3cdaa-34b6-4684-87fd-d9eeb10e4905 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:11.245426 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd3cdaa-34b6-4684-87fd-d9eeb10e4905 admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:11.246237 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd3cdaa-34b6-4684-87fd-d9eeb10e4905 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:11.263619 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd3cdaa-34b6-4684-87fd-d9eeb10e4905 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:11.318386 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 954/1883] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:26:11 2022] POST /identity/v3/users => generated 294 bytes in 302 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:26:11.322665 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560b286e-41f7-4236-a07a-ec4e1771b51b admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:11.338762 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560b286e-41f7-4236-a07a-ec4e1771b51b admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:11.346509 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97964b73-123e-4ac1-a264-743ac3e6b4e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:11.348912 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97964b73-123e-4ac1-a264-743ac3e6b4e2 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:11.349478 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97964b73-123e-4ac1-a264-743ac3e6b4e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:11.350084 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97964b73-123e-4ac1-a264-743ac3e6b4e2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:11.351287 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97964b73-123e-4ac1-a264-743ac3e6b4e2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:11.353756 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97964b73-123e-4ac1-a264-743ac3e6b4e2 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:11.374937 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 930/1884] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:26:11 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:26:11.380851 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd3cdaa-34b6-4684-87fd-d9eeb10e4905 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:11.397268 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd3cdaa-34b6-4684-87fd-d9eeb10e4905 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:11.443587 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4c421e-60de-4e87-b770-3f3880f4e558 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:11.447936 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4c421e-60de-4e87-b770-3f3880f4e558 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:11.448549 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4c421e-60de-4e87-b770-3f3880f4e558 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:11.448889 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4c421e-60de-4e87-b770-3f3880f4e558 admin admin] PATH_INFO: `/v3/projects/698e3b45fa3543c58665717e84d251ea/users/8871f883f18f409cbb34817e7b80896c/roles/414286927fdd44688107ff5e10235acf` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:11.487281 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c4c421e-60de-4e87-b770-3f3880f4e558 admin admin] RBAC: Authorizing `identity:create_grant(project_id=698e3b45fa3543c58665717e84d251ea, user_id=8871f883f18f409cbb34817e7b80896c, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:11.510279 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c4c421e-60de-4e87-b770-3f3880f4e558 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:11.560109 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 955/1885] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:26:11 2022] PUT /identity/v3/projects/698e3b45fa3543c58665717e84d251ea/users/8871f883f18f409cbb34817e7b80896c/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:11.564712 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97964b73-123e-4ac1-a264-743ac3e6b4e2 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:11.589011 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97964b73-123e-4ac1-a264-743ac3e6b4e2 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:11.700966 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:11.709483 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69e9787-268f-4cf7-82c0-2abdc502bc8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:11.712114 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69e9787-268f-4cf7-82c0-2abdc502bc8d admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:11.712432 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69e9787-268f-4cf7-82c0-2abdc502bc8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:11.712717 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69e9787-268f-4cf7-82c0-2abdc502bc8d admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:11.713445 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e69e9787-268f-4cf7-82c0-2abdc502bc8d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:11.715833 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e69e9787-268f-4cf7-82c0-2abdc502bc8d admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:11.743178 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 931/1886] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:26:11 2022] GET /identity/v3/roles => generated 1420 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:26:11.747230 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4c421e-60de-4e87-b770-3f3880f4e558 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:11.760894 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4c421e-60de-4e87-b770-3f3880f4e558 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:11.778888 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033ffeb7-9c9f-4f72-8912-a05f47b86d4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:11.779553 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033ffeb7-9c9f-4f72-8912-a05f47b86d4c admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:11.779812 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033ffeb7-9c9f-4f72-8912-a05f47b86d4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:11.780077 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033ffeb7-9c9f-4f72-8912-a05f47b86d4c admin admin] PATH_INFO: `/v3/projects/698e3b45fa3543c58665717e84d251ea/users/8871f883f18f409cbb34817e7b80896c/roles/414286927fdd44688107ff5e10235acf` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:11.782456 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033ffeb7-9c9f-4f72-8912-a05f47b86d4c admin admin] RBAC: Authorizing `identity:create_grant(project_id=698e3b45fa3543c58665717e84d251ea, user_id=8871f883f18f409cbb34817e7b80896c, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:11.786267 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033ffeb7-9c9f-4f72-8912-a05f47b86d4c admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:11.819022 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 956/1887] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:26:11 2022] PUT /identity/v3/projects/698e3b45fa3543c58665717e84d251ea/users/8871f883f18f409cbb34817e7b80896c/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:11.830226 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cbe3aa-613d-40fc-bd05-eabacc0b9f56 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:11.831014 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cbe3aa-613d-40fc-bd05-eabacc0b9f56 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:11.831784 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cbe3aa-613d-40fc-bd05-eabacc0b9f56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:11.897739 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-b4cbe3aa-613d-40fc-bd05-eabacc0b9f56 None None] MFA Rules not processed for user `8871f883f18f409cbb34817e7b80896c`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:11.899943 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-b4cbe3aa-613d-40fc-bd05-eabacc0b9f56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:12.062307 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 932/1888] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:11 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:12.075623 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033ffeb7-9c9f-4f72-8912-a05f47b86d4c admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:12.078245 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-033ffeb7-9c9f-4f72-8912-a05f47b86d4c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:12.202380 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:12.220178 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033ffeb7-9c9f-4f72-8912-a05f47b86d4c admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:12.234940 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b981ba-4ae3-4b8e-9ffe-54cd3ee64332 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:12.235583 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b981ba-4ae3-4b8e-9ffe-54cd3ee64332 service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:12.235919 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b981ba-4ae3-4b8e-9ffe-54cd3ee64332 service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:12.236110 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b981ba-4ae3-4b8e-9ffe-54cd3ee64332 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:12.360834 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4cbe3aa-613d-40fc-bd05-eabacc0b9f56 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:12.364246 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-b4cbe3aa-613d-40fc-bd05-eabacc0b9f56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:12.404319 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b981ba-4ae3-4b8e-9ffe-54cd3ee64332 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:12.407552 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b981ba-4ae3-4b8e-9ffe-54cd3ee64332 service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:12.450850 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 957/1889] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:26:12 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:26:12.536582 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4cbe3aa-613d-40fc-bd05-eabacc0b9f56 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:12.548705 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37935223-58e0-4b94-9087-b09aa466d133 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:12.549361 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37935223-58e0-4b94-9087-b09aa466d133 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:12.549615 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37935223-58e0-4b94-9087-b09aa466d133 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:12.549926 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37935223-58e0-4b94-9087-b09aa466d133 admin admin] PATH_INFO: `/v3/users/2ac8bc3fb2734e979ae50aff1a6e905e` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:12.553218 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37935223-58e0-4b94-9087-b09aa466d133 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2ac8bc3fb2734e979ae50aff1a6e905e)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:12.555423 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37935223-58e0-4b94-9087-b09aa466d133 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:12.678323 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-37935223-58e0-4b94-9087-b09aa466d133 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2ac8bc3fb2734e979ae50aff1a6e905e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:12.709114 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-37935223-58e0-4b94-9087-b09aa466d133 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2ac8bc3fb2734e979ae50aff1a6e905e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:12.710953 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-37935223-58e0-4b94-9087-b09aa466d133 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2ac8bc3fb2734e979ae50aff1a6e905e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:12.758632 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-37935223-58e0-4b94-9087-b09aa466d133 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2ac8bc3fb2734e979ae50aff1a6e905e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:12.770726 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:12.777748 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-37935223-58e0-4b94-9087-b09aa466d133 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2ac8bc3fb2734e979ae50aff1a6e905e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:12.796963 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 933/1890] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:26:12 2022] DELETE /identity/v3/users/2ac8bc3fb2734e979ae50aff1a6e905e => generated 0 bytes in 438 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:12.802953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc669c6-15e7-454c-a0f1-25120a39debb None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:12.803304 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc669c6-15e7-454c-a0f1-25120a39debb None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:12.803601 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc669c6-15e7-454c-a0f1-25120a39debb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:12.852466 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-bdc669c6-15e7-454c-a0f1-25120a39debb None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:12.869134 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-bdc669c6-15e7-454c-a0f1-25120a39debb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:13.012334 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 958/1891] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:13.087390 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37935223-58e0-4b94-9087-b09aa466d133 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:13.088888 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-37935223-58e0-4b94-9087-b09aa466d133 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:13.242400 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37935223-58e0-4b94-9087-b09aa466d133 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:13.251206 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427c8fe5-988e-480d-b37d-e0d19d476abb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:13.251852 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427c8fe5-988e-480d-b37d-e0d19d476abb service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:13.252136 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427c8fe5-988e-480d-b37d-e0d19d476abb service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:13.253124 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427c8fe5-988e-480d-b37d-e0d19d476abb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:13.264614 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427c8fe5-988e-480d-b37d-e0d19d476abb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:13.267731 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427c8fe5-988e-480d-b37d-e0d19d476abb service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:13.280827 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 934/1892] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:26:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:26:13.390545 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615e638c-fff7-4214-9746-6578dab2c47d None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:13.391184 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615e638c-fff7-4214-9746-6578dab2c47d None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:13.391663 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615e638c-fff7-4214-9746-6578dab2c47d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:13.428251 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-615e638c-fff7-4214-9746-6578dab2c47d None None] MFA Rules not processed for user `9f0842753cea48fe9508de4fc4d15b5f`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:13.429950 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-615e638c-fff7-4214-9746-6578dab2c47d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:13.543879 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 959/1893] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:13 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:13.560202 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427c8fe5-988e-480d-b37d-e0d19d476abb service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:13.561959 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-427c8fe5-988e-480d-b37d-e0d19d476abb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:13.664529 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427c8fe5-988e-480d-b37d-e0d19d476abb service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:13.673979 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1695b9-6f36-4ab2-98c3-75090fcdb0ff service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:13.677202 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1695b9-6f36-4ab2-98c3-75090fcdb0ff service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:13.677562 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1695b9-6f36-4ab2-98c3-75090fcdb0ff service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:13.677808 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1695b9-6f36-4ab2-98c3-75090fcdb0ff service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:13.679903 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-5d1695b9-6f36-4ab2-98c3-75090fcdb0ff service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:13.762923 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1695b9-6f36-4ab2-98c3-75090fcdb0ff service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:13.765175 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1695b9-6f36-4ab2-98c3-75090fcdb0ff service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:13.790247 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 935/1894] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:26:13 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 230 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:26:14.033646 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-615e638c-fff7-4214-9746-6578dab2c47d None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:14.036459 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-615e638c-fff7-4214-9746-6578dab2c47d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:14.109692 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-615e638c-fff7-4214-9746-6578dab2c47d None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:14.127540 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6664f38e-7cc9-4f3b-b6b1-aa1207a08dea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:14.129085 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6664f38e-7cc9-4f3b-b6b1-aa1207a08dea admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:14.130917 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6664f38e-7cc9-4f3b-b6b1-aa1207a08dea admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:14.133665 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6664f38e-7cc9-4f3b-b6b1-aa1207a08dea admin admin] PATH_INFO: `/v3/projects/f135389d386e43d58581df01f80c083c` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:14.142784 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6664f38e-7cc9-4f3b-b6b1-aa1207a08dea admin admin] RBAC: Authorizing `identity:delete_project(project_id=f135389d386e43d58581df01f80c083c)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:14.147492 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6664f38e-7cc9-4f3b-b6b1-aa1207a08dea admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:14.193048 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:14.283387 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-6664f38e-7cc9-4f3b-b6b1-aa1207a08dea admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f135389d386e43d58581df01f80c083c', 'request_id': 'req-6664f38e-7cc9-4f3b-b6b1-aa1207a08dea'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:14.302392 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-6664f38e-7cc9-4f3b-b6b1-aa1207a08dea admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f135389d386e43d58581df01f80c083c', 'request_id': 'req-6664f38e-7cc9-4f3b-b6b1-aa1207a08dea'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:14.310780 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-6664f38e-7cc9-4f3b-b6b1-aa1207a08dea admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f135389d386e43d58581df01f80c083c', 'request_id': 'req-6664f38e-7cc9-4f3b-b6b1-aa1207a08dea'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:14.324713 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-6664f38e-7cc9-4f3b-b6b1-aa1207a08dea admin admin] The token cache is being invalidate because project f135389d386e43d58581df01f80c083c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:26:14.325323 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-6664f38e-7cc9-4f3b-b6b1-aa1207a08dea admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:14.325909 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-6664f38e-7cc9-4f3b-b6b1-aa1207a08dea admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:14.331543 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 960/1895] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:26:14 2022] DELETE /identity/v3/projects/f135389d386e43d58581df01f80c083c => generated 0 bytes in 295 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:14.333641 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae1ec15-5286-4ab6-877e-2f858386de87 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:14.333925 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae1ec15-5286-4ab6-877e-2f858386de87 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:14.334610 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae1ec15-5286-4ab6-877e-2f858386de87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:14.380135 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-0ae1ec15-5286-4ab6-877e-2f858386de87 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:14.386276 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0ae1ec15-5286-4ab6-877e-2f858386de87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:14.461585 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:14.562222 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 936/1896] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:14.566358 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39af36ee-c499-432a-9f45-0edb5bf5d06a None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:14.566810 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39af36ee-c499-432a-9f45-0edb5bf5d06a None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:14.566810 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39af36ee-c499-432a-9f45-0edb5bf5d06a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:14.606793 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-39af36ee-c499-432a-9f45-0edb5bf5d06a None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:14.608041 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-39af36ee-c499-432a-9f45-0edb5bf5d06a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:14.631681 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 961/1897] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 66 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:14.636413 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae1ec15-5286-4ab6-877e-2f858386de87 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:14.660309 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39af36ee-c499-432a-9f45-0edb5bf5d06a None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:14.661312 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae1ec15-5286-4ab6-877e-2f858386de87 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:14.662657 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-39af36ee-c499-432a-9f45-0edb5bf5d06a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:14.677785 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d6c8e0-e6d7-4483-9082-5683a4914de6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:14.678692 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d6c8e0-e6d7-4483-9082-5683a4914de6 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:14.679043 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d6c8e0-e6d7-4483-9082-5683a4914de6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:14.680147 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d6c8e0-e6d7-4483-9082-5683a4914de6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:14.680330 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d6c8e0-e6d7-4483-9082-5683a4914de6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:26:14.680878 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d6c8e0-e6d7-4483-9082-5683a4914de6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:14.682600 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d6c8e0-e6d7-4483-9082-5683a4914de6 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:14.715253 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 937/1898] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:26:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:26:14.717194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d6c8e0-e6d7-4483-9082-5683a4914de6 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:14.746493 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d6c8e0-e6d7-4483-9082-5683a4914de6 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:14.771351 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e834ab7f-ecc4-4c0e-bcd4-edfd49225acd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:14.771973 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e834ab7f-ecc4-4c0e-bcd4-edfd49225acd admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:14.775365 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e834ab7f-ecc4-4c0e-bcd4-edfd49225acd admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:14.775824 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e834ab7f-ecc4-4c0e-bcd4-edfd49225acd admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:14.776643 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e834ab7f-ecc4-4c0e-bcd4-edfd49225acd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:14.778788 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e834ab7f-ecc4-4c0e-bcd4-edfd49225acd admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:14.813114 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39af36ee-c499-432a-9f45-0edb5bf5d06a None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:14.824672 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a24c978-1b37-40f2-a9bb-f46c4b7075f6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:14.825665 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a24c978-1b37-40f2-a9bb-f46c4b7075f6 service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:14.831947 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a24c978-1b37-40f2-a9bb-f46c4b7075f6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:14.834813 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a24c978-1b37-40f2-a9bb-f46c4b7075f6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:14.839224 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-7a24c978-1b37-40f2-a9bb-f46c4b7075f6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:14.884892 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 938/1899] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:26:14 2022] POST /identity/v3/projects => generated 399 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:26:14.888643 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e834ab7f-ecc4-4c0e-bcd4-edfd49225acd admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:14.978461 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a24c978-1b37-40f2-a9bb-f46c4b7075f6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:14.981820 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a24c978-1b37-40f2-a9bb-f46c4b7075f6 service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:15.005441 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 962/1900] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:26:14 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 27 01:26:15.039214 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e834ab7f-ecc4-4c0e-bcd4-edfd49225acd admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:15.050576 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e530c11-8080-4cb2-958b-74379a655968 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:15.051199 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e530c11-8080-4cb2-958b-74379a655968 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:15.051475 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e530c11-8080-4cb2-958b-74379a655968 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:15.051746 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e530c11-8080-4cb2-958b-74379a655968 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:15.052506 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e530c11-8080-4cb2-958b-74379a655968 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:15.054317 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e530c11-8080-4cb2-958b-74379a655968 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:15.086300 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 939/1901] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:26:14 2022] POST /identity/v3/users => generated 300 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:26:15.088240 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a24c978-1b37-40f2-a9bb-f46c4b7075f6 service cinder] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:15.105107 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a24c978-1b37-40f2-a9bb-f46c4b7075f6 service cinder] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:15.119149 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0247e62-54ea-4db6-98ce-21c1f7c3f39a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:15.119774 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0247e62-54ea-4db6-98ce-21c1f7c3f39a admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:15.120060 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0247e62-54ea-4db6-98ce-21c1f7c3f39a admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:15.120337 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0247e62-54ea-4db6-98ce-21c1f7c3f39a admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:15.120990 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0247e62-54ea-4db6-98ce-21c1f7c3f39a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:15.122659 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0247e62-54ea-4db6-98ce-21c1f7c3f39a admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:15.148227 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 963/1902] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:26:15 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:26:15.152682 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e530c11-8080-4cb2-958b-74379a655968 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:15.167696 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e530c11-8080-4cb2-958b-74379a655968 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:15.174354 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c0ba937-9918-4a52-8a8b-cab68dd83466 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:15.178205 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c0ba937-9918-4a52-8a8b-cab68dd83466 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:15.179998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c0ba937-9918-4a52-8a8b-cab68dd83466 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:15.180572 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c0ba937-9918-4a52-8a8b-cab68dd83466 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:15.181712 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c0ba937-9918-4a52-8a8b-cab68dd83466 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:15.184100 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c0ba937-9918-4a52-8a8b-cab68dd83466 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:15.206506 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 940/1903] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:26:15 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:26:15.207878 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0247e62-54ea-4db6-98ce-21c1f7c3f39a admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:15.223368 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0247e62-54ea-4db6-98ce-21c1f7c3f39a admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:15.231062 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c12501-817b-4b85-bd96-a37858f19f92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:15.232024 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c12501-817b-4b85-bd96-a37858f19f92 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:15.232523 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c12501-817b-4b85-bd96-a37858f19f92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:15.233030 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c12501-817b-4b85-bd96-a37858f19f92 admin admin] PATH_INFO: `/v3/projects/a5b460240ea649b0b12d8b9387003dc8/users/2c97470658cb463c838d494d1e4ad2a8/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:15.257570 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c12501-817b-4b85-bd96-a37858f19f92 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a5b460240ea649b0b12d8b9387003dc8, user_id=2c97470658cb463c838d494d1e4ad2a8, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:15.261931 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c12501-817b-4b85-bd96-a37858f19f92 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:15.286506 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 964/1904] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:26:15 2022] PUT /identity/v3/projects/a5b460240ea649b0b12d8b9387003dc8/users/2c97470658cb463c838d494d1e4ad2a8/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 80 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:15.301459 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de50d089-b124-4e1a-9c52-495ccd3a2605 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:15.301819 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de50d089-b124-4e1a-9c52-495ccd3a2605 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:15.302080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de50d089-b124-4e1a-9c52-495ccd3a2605 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:15.344215 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-de50d089-b124-4e1a-9c52-495ccd3a2605 None None] MFA Rules not processed for user `2c97470658cb463c838d494d1e4ad2a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:15.348033 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-de50d089-b124-4e1a-9c52-495ccd3a2605 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:15.381366 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b38217-85da-4065-819d-e5b611e32c7c None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:15.382053 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b38217-85da-4065-819d-e5b611e32c7c None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:15.382939 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b38217-85da-4065-819d-e5b611e32c7c None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:15.389388 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 965/1905] 10.222.0.42 () {54 vars in 921 bytes} [Thu Oct 27 01:26:15 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:26:15.444499 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5d5ca8-f006-4ea1-b9aa-e2356cc6bbad None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:15.444886 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5d5ca8-f006-4ea1-b9aa-e2356cc6bbad None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:15.445501 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5d5ca8-f006-4ea1-b9aa-e2356cc6bbad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:15.497678 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-dd5d5ca8-f006-4ea1-b9aa-e2356cc6bbad None None] MFA Rules not processed for user `62051562fc0b4cd08750d2293e71787c`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:15.501254 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-dd5d5ca8-f006-4ea1-b9aa-e2356cc6bbad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:15.502081 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 941/1906] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:15 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:15.525481 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de50d089-b124-4e1a-9c52-495ccd3a2605 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:15.529444 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-de50d089-b124-4e1a-9c52-495ccd3a2605 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:15.536874 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:15.572968 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:15.636087 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 966/1907] 10.222.0.42 () {58 vars in 1018 bytes} [Thu Oct 27 01:26:15 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:26:15.637324 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5d5ca8-f006-4ea1-b9aa-e2356cc6bbad None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:15.640362 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-dd5d5ca8-f006-4ea1-b9aa-e2356cc6bbad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:15.718334 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de50d089-b124-4e1a-9c52-495ccd3a2605 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:15.729766 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a78ba76-3ba0-4534-b229-4db0301a9e78 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:15.730223 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a78ba76-3ba0-4534-b229-4db0301a9e78 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:15.730667 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a78ba76-3ba0-4534-b229-4db0301a9e78 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:15.730950 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a78ba76-3ba0-4534-b229-4db0301a9e78 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:15.750863 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a78ba76-3ba0-4534-b229-4db0301a9e78 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:15.756707 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a78ba76-3ba0-4534-b229-4db0301a9e78 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:15.778531 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 942/1908] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:26:15 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 254 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:26:15.779490 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a78ba76-3ba0-4534-b229-4db0301a9e78 service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:15.793150 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5d5ca8-f006-4ea1-b9aa-e2356cc6bbad None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:15.796326 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-3a78ba76-3ba0-4534-b229-4db0301a9e78 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:15.833540 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a76a23-49db-4cc6-a1cc-f55619513db3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:15.834424 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a76a23-49db-4cc6-a1cc-f55619513db3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:15.834721 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a76a23-49db-4cc6-a1cc-f55619513db3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:15.834992 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a76a23-49db-4cc6-a1cc-f55619513db3 admin admin] PATH_INFO: `/v3/users/68909cf0c5bf4875927b30d9a6ce8cd9` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:15.843793 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a76a23-49db-4cc6-a1cc-f55619513db3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=68909cf0c5bf4875927b30d9a6ce8cd9)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:15.853527 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a76a23-49db-4cc6-a1cc-f55619513db3 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:15.855895 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a78ba76-3ba0-4534-b229-4db0301a9e78 service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:15.877195 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2149deb5-dd90-4d94-aa85-073167ec6c53 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62051562fc0b4cd08750d2293e71787c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:15.878363 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2149deb5-dd90-4d94-aa85-073167ec6c53 service placement] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:15.880077 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2149deb5-dd90-4d94-aa85-073167ec6c53 service placement] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:15.880077 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2149deb5-dd90-4d94-aa85-073167ec6c53 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:15.905550 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2149deb5-dd90-4d94-aa85-073167ec6c53 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:15.910422 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2149deb5-dd90-4d94-aa85-073167ec6c53 service placement] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:15.975049 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 943/1909] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:26:15 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 196 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:26:15.976278 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2149deb5-dd90-4d94-aa85-073167ec6c53 service placement] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:15.998343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-e7a76a23-49db-4cc6-a1cc-f55619513db3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '68909cf0c5bf4875927b30d9a6ce8cd9'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:16.007947 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2149deb5-dd90-4d94-aa85-073167ec6c53 service placement] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:16.017197 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-e7a76a23-49db-4cc6-a1cc-f55619513db3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '68909cf0c5bf4875927b30d9a6ce8cd9'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:16.029218 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f2fb145-cde1-4d95-bbaf-8e90ac19441a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:16.034047 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2fb145-cde1-4d95-bbaf-8e90ac19441a service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:16.034878 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2fb145-cde1-4d95-bbaf-8e90ac19441a service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:16.036255 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2fb145-cde1-4d95-bbaf-8e90ac19441a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:16.043405 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-1f2fb145-cde1-4d95-bbaf-8e90ac19441a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:16.060867 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-e7a76a23-49db-4cc6-a1cc-f55619513db3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '68909cf0c5bf4875927b30d9a6ce8cd9'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:16.064894 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-e7a76a23-49db-4cc6-a1cc-f55619513db3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '68909cf0c5bf4875927b30d9a6ce8cd9'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:16.067833 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-e7a76a23-49db-4cc6-a1cc-f55619513db3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '68909cf0c5bf4875927b30d9a6ce8cd9'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:16.104984 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 967/1910] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:26:15 2022] DELETE /identity/v3/users/68909cf0c5bf4875927b30d9a6ce8cd9 => generated 0 bytes in 468 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:16.187439 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f2fb145-cde1-4d95-bbaf-8e90ac19441a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:16.188955 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f2fb145-cde1-4d95-bbaf-8e90ac19441a service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:16.202050 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 944/1911] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:26:15 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 226 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:26:16.474245 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a76a23-49db-4cc6-a1cc-f55619513db3 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:16.477617 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-e7a76a23-49db-4cc6-a1cc-f55619513db3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:16.587568 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:16.673437 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a76a23-49db-4cc6-a1cc-f55619513db3 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:16.712605 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7557b0f-c683-4476-8417-591f283367c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:16.713846 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7557b0f-c683-4476-8417-591f283367c9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:16.714355 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7557b0f-c683-4476-8417-591f283367c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:16.714826 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7557b0f-c683-4476-8417-591f283367c9 admin admin] PATH_INFO: `/v3/projects/c72a2266373f44749fa0d3bbd3bb68cd` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:16.716221 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7557b0f-c683-4476-8417-591f283367c9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c72a2266373f44749fa0d3bbd3bb68cd)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:16.719643 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7557b0f-c683-4476-8417-591f283367c9 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:16.785091 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-b7557b0f-c683-4476-8417-591f283367c9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c72a2266373f44749fa0d3bbd3bb68cd', 'request_id': 'req-b7557b0f-c683-4476-8417-591f283367c9'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:16.800352 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-b7557b0f-c683-4476-8417-591f283367c9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c72a2266373f44749fa0d3bbd3bb68cd', 'request_id': 'req-b7557b0f-c683-4476-8417-591f283367c9'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:16.808970 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-b7557b0f-c683-4476-8417-591f283367c9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c72a2266373f44749fa0d3bbd3bb68cd', 'request_id': 'req-b7557b0f-c683-4476-8417-591f283367c9'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:16.838592 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-b7557b0f-c683-4476-8417-591f283367c9 admin admin] The token cache is being invalidate because project c72a2266373f44749fa0d3bbd3bb68cd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:26:16.838886 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-b7557b0f-c683-4476-8417-591f283367c9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:16.839983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-b7557b0f-c683-4476-8417-591f283367c9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:16.841731 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 968/1912] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:26:16 2022] DELETE /identity/v3/projects/c72a2266373f44749fa0d3bbd3bb68cd => generated 0 bytes in 368 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:16.846922 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16834878-3501-4242-9f17-9f2d6603fa7e None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:16.848250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16834878-3501-4242-9f17-9f2d6603fa7e None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:16.848250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16834878-3501-4242-9f17-9f2d6603fa7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:16.870769 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-16834878-3501-4242-9f17-9f2d6603fa7e None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:16.872010 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-16834878-3501-4242-9f17-9f2d6603fa7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:16.947133 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce2e1a7-e6b9-4834-8baf-1dcf6dd6d1d2 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:16.947419 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce2e1a7-e6b9-4834-8baf-1dcf6dd6d1d2 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:16.948136 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce2e1a7-e6b9-4834-8baf-1dcf6dd6d1d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:16.975357 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-9ce2e1a7-e6b9-4834-8baf-1dcf6dd6d1d2 None None] MFA Rules not processed for user `181df03f4bba4f96afc3c888b3ca9b4d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:16.976957 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-9ce2e1a7-e6b9-4834-8baf-1dcf6dd6d1d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:17.000317 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 945/1913] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:17.009462 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd36ccef-2d09-4424-91a1-d8c4ee83e084 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:17.009890 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd36ccef-2d09-4424-91a1-d8c4ee83e084 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:17.010339 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd36ccef-2d09-4424-91a1-d8c4ee83e084 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:17.031597 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-fd36ccef-2d09-4424-91a1-d8c4ee83e084 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:17.036464 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-fd36ccef-2d09-4424-91a1-d8c4ee83e084 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:17.046077 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 946/1914] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:26:17.050114 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd36ccef-2d09-4424-91a1-d8c4ee83e084 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:17.060752 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd36ccef-2d09-4424-91a1-d8c4ee83e084 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:17.066430 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 969/1915] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:16 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:17.069463 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5964faa-9c8f-46df-b150-a052c89b212a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:17.070283 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5964faa-9c8f-46df-b150-a052c89b212a admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:17.070778 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5964faa-9c8f-46df-b150-a052c89b212a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:17.070944 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5964faa-9c8f-46df-b150-a052c89b212a admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:17.072091 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5964faa-9c8f-46df-b150-a052c89b212a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:26:17.072615 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5964faa-9c8f-46df-b150-a052c89b212a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:17.074412 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5964faa-9c8f-46df-b150-a052c89b212a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:17.089767 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 947/1916] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:26:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:26:17.094281 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce2e1a7-e6b9-4834-8baf-1dcf6dd6d1d2 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:17.110496 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce2e1a7-e6b9-4834-8baf-1dcf6dd6d1d2 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:17.116138 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5bc4e2-60dd-48f7-b42d-52db5cec1eab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:17.116768 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5bc4e2-60dd-48f7-b42d-52db5cec1eab admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:17.117089 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5bc4e2-60dd-48f7-b42d-52db5cec1eab admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:17.117380 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5bc4e2-60dd-48f7-b42d-52db5cec1eab admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:17.118104 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e5bc4e2-60dd-48f7-b42d-52db5cec1eab admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:17.120028 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e5bc4e2-60dd-48f7-b42d-52db5cec1eab admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:17.158532 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 970/1917] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:26:17 2022] POST /identity/v3/projects => generated 405 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:26:17.160975 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5964faa-9c8f-46df-b150-a052c89b212a admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:17.174514 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5964faa-9c8f-46df-b150-a052c89b212a admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:17.186308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:17.243071 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f0b1ff-532d-4843-b14c-57179b9e100a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:17.243692 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f0b1ff-532d-4843-b14c-57179b9e100a admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:17.243966 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f0b1ff-532d-4843-b14c-57179b9e100a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:17.244282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f0b1ff-532d-4843-b14c-57179b9e100a admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:17.245030 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f0b1ff-532d-4843-b14c-57179b9e100a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:17.247203 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f0b1ff-532d-4843-b14c-57179b9e100a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:17.274743 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 948/1918] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:26:17 2022] POST /identity/v3/users => generated 303 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:26:17.279047 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5bc4e2-60dd-48f7-b42d-52db5cec1eab admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:17.290579 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5bc4e2-60dd-48f7-b42d-52db5cec1eab admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:17.296104 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52cf4a1-e9ba-4dbc-a366-2232bb212aa4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:17.296961 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52cf4a1-e9ba-4dbc-a366-2232bb212aa4 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:17.297520 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52cf4a1-e9ba-4dbc-a366-2232bb212aa4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:17.298166 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52cf4a1-e9ba-4dbc-a366-2232bb212aa4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:17.299684 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a52cf4a1-e9ba-4dbc-a366-2232bb212aa4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:17.302015 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a52cf4a1-e9ba-4dbc-a366-2232bb212aa4 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:17.317106 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 971/1919] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:26:17 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:26:17.321562 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f0b1ff-532d-4843-b14c-57179b9e100a admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:17.337541 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f0b1ff-532d-4843-b14c-57179b9e100a admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:17.343633 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e362bb6c-f356-49ea-8f81-77dc80f306bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:17.344354 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e362bb6c-f356-49ea-8f81-77dc80f306bb admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:17.344700 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e362bb6c-f356-49ea-8f81-77dc80f306bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:17.345039 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e362bb6c-f356-49ea-8f81-77dc80f306bb admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:17.347490 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e362bb6c-f356-49ea-8f81-77dc80f306bb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:17.349285 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e362bb6c-f356-49ea-8f81-77dc80f306bb admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:17.385228 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 949/1920] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:26:17 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:26:17.388922 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52cf4a1-e9ba-4dbc-a366-2232bb212aa4 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:17.408906 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52cf4a1-e9ba-4dbc-a366-2232bb212aa4 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:17.421982 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7515cff7-eef3-4ae1-a747-c9b88478e01f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:17.422744 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7515cff7-eef3-4ae1-a747-c9b88478e01f admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:17.423031 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7515cff7-eef3-4ae1-a747-c9b88478e01f admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:17.423301 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7515cff7-eef3-4ae1-a747-c9b88478e01f admin admin] PATH_INFO: `/v3/projects/23904d82da0244c0aaa0523d563e2c90/users/4f619ab4b6e94d23bd883ef3b4b96c14/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:17.456183 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7515cff7-eef3-4ae1-a747-c9b88478e01f admin admin] RBAC: Authorizing `identity:create_grant(project_id=23904d82da0244c0aaa0523d563e2c90, user_id=4f619ab4b6e94d23bd883ef3b4b96c14, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:17.465944 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7515cff7-eef3-4ae1-a747-c9b88478e01f admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:17.493409 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 972/1921] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:26:17 2022] PUT /identity/v3/projects/23904d82da0244c0aaa0523d563e2c90/users/4f619ab4b6e94d23bd883ef3b4b96c14/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:17.505940 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5baa82a2-b183-4ca4-9e8f-faa299bc665a None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:17.506372 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5baa82a2-b183-4ca4-9e8f-faa299bc665a None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:17.506749 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5baa82a2-b183-4ca4-9e8f-faa299bc665a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:17.543909 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-5baa82a2-b183-4ca4-9e8f-faa299bc665a None None] MFA Rules not processed for user `4f619ab4b6e94d23bd883ef3b4b96c14`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:17.545799 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-5baa82a2-b183-4ca4-9e8f-faa299bc665a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:17.663855 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 950/1922] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:17 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:26:17.665424 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7515cff7-eef3-4ae1-a747-c9b88478e01f admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:17.683075 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:17.746990 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7515cff7-eef3-4ae1-a747-c9b88478e01f admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:17.755990 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d234f200-96f9-4fd3-a909-5886860a1328 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:17.756624 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d234f200-96f9-4fd3-a909-5886860a1328 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:17.756906 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d234f200-96f9-4fd3-a909-5886860a1328 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:17.757283 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d234f200-96f9-4fd3-a909-5886860a1328 admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:17.757984 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d234f200-96f9-4fd3-a909-5886860a1328 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:17.759897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d234f200-96f9-4fd3-a909-5886860a1328 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:17.799647 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 973/1923] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:26:17 2022] POST /identity/v3/projects => generated 405 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:26:17.804103 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5baa82a2-b183-4ca4-9e8f-faa299bc665a None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:17.822846 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5baa82a2-b183-4ca4-9e8f-faa299bc665a None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:17.960788 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1752fe34-11bb-4cd9-8b8a-095db2e18246 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:17.962868 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1752fe34-11bb-4cd9-8b8a-095db2e18246 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:17.963502 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1752fe34-11bb-4cd9-8b8a-095db2e18246 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:17.964401 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1752fe34-11bb-4cd9-8b8a-095db2e18246 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:17.966011 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1752fe34-11bb-4cd9-8b8a-095db2e18246 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:17.970085 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1752fe34-11bb-4cd9-8b8a-095db2e18246 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:18.004849 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 951/1924] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:26:17 2022] POST /identity/v3/users => generated 309 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:26:18.010445 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d234f200-96f9-4fd3-a909-5886860a1328 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:18.030112 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d234f200-96f9-4fd3-a909-5886860a1328 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:18.038591 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e092de6-3508-49b4-8059-9bd0ececf90b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:18.039203 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e092de6-3508-49b4-8059-9bd0ececf90b admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:18.039613 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e092de6-3508-49b4-8059-9bd0ececf90b admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:18.040073 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e092de6-3508-49b4-8059-9bd0ececf90b admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:18.042247 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e092de6-3508-49b4-8059-9bd0ececf90b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:18.044266 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e092de6-3508-49b4-8059-9bd0ececf90b admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:18.064119 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 974/1925] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:26:18 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:26:18.069043 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1752fe34-11bb-4cd9-8b8a-095db2e18246 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:18.082726 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1752fe34-11bb-4cd9-8b8a-095db2e18246 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:18.090278 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db69dcd-6594-4f10-9438-77d5861511d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:18.090978 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db69dcd-6594-4f10-9438-77d5861511d6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:18.091259 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db69dcd-6594-4f10-9438-77d5861511d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:18.092948 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db69dcd-6594-4f10-9438-77d5861511d6 admin admin] PATH_INFO: `/v3/projects/a68e8a327fd34d7a84dcf1d73112a917/users/0e6e509a188849ed8ed3d92911588d35/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:18.114090 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:18.131162 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7db69dcd-6594-4f10-9438-77d5861511d6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a68e8a327fd34d7a84dcf1d73112a917, user_id=0e6e509a188849ed8ed3d92911588d35, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:18.136542 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7db69dcd-6594-4f10-9438-77d5861511d6 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:18.183238 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e092de6-3508-49b4-8059-9bd0ececf90b admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:18.183839 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 952/1926] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:26:18 2022] PUT /identity/v3/projects/a68e8a327fd34d7a84dcf1d73112a917/users/0e6e509a188849ed8ed3d92911588d35/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:18.198342 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e092de6-3508-49b4-8059-9bd0ececf90b admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:18.284588 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5efef7-065d-4173-94b8-5d59958e28b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:18.285613 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5efef7-065d-4173-94b8-5d59958e28b9 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:18.285952 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5efef7-065d-4173-94b8-5d59958e28b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:18.286368 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5efef7-065d-4173-94b8-5d59958e28b9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:18.287568 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5efef7-065d-4173-94b8-5d59958e28b9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:18.289102 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5efef7-065d-4173-94b8-5d59958e28b9 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:18.301858 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 975/1927] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:26:18 2022] GET /identity/v3/roles => generated 1420 bytes in 119 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:26:18.306028 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db69dcd-6594-4f10-9438-77d5861511d6 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:18.324904 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db69dcd-6594-4f10-9438-77d5861511d6 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:18.334899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db4e6416-8a20-4306-8c4b-ca98ea6c4403 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:18.335995 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4e6416-8a20-4306-8c4b-ca98ea6c4403 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:18.336414 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4e6416-8a20-4306-8c4b-ca98ea6c4403 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:18.336870 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4e6416-8a20-4306-8c4b-ca98ea6c4403 admin admin] PATH_INFO: `/v3/projects/a68e8a327fd34d7a84dcf1d73112a917/users/0e6e509a188849ed8ed3d92911588d35/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:18.339491 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db4e6416-8a20-4306-8c4b-ca98ea6c4403 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a68e8a327fd34d7a84dcf1d73112a917, user_id=0e6e509a188849ed8ed3d92911588d35, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:18.342718 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db4e6416-8a20-4306-8c4b-ca98ea6c4403 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:18.372023 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 953/1928] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:26:18 2022] PUT /identity/v3/projects/a68e8a327fd34d7a84dcf1d73112a917/users/0e6e509a188849ed8ed3d92911588d35/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:18.384173 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8e5215-5980-45d1-ab92-495b5ee1a96a None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:18.390469 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8e5215-5980-45d1-ab92-495b5ee1a96a None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:18.391051 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8e5215-5980-45d1-ab92-495b5ee1a96a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:18.447520 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-1c8e5215-5980-45d1-ab92-495b5ee1a96a None None] MFA Rules not processed for user `0e6e509a188849ed8ed3d92911588d35`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:18.449357 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-1c8e5215-5980-45d1-ab92-495b5ee1a96a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:18.599316 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 976/1929] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:18 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 216 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:18.613635 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db4e6416-8a20-4306-8c4b-ca98ea6c4403 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:18.615558 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-db4e6416-8a20-4306-8c4b-ca98ea6c4403 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:18.735835 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db4e6416-8a20-4306-8c4b-ca98ea6c4403 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:18.740214 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9efde03f-a94d-4073-8572-d48c22cb863d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:18.740834 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9efde03f-a94d-4073-8572-d48c22cb863d service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:18.741139 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9efde03f-a94d-4073-8572-d48c22cb863d service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:18.741412 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9efde03f-a94d-4073-8572-d48c22cb863d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:18.751924 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9efde03f-a94d-4073-8572-d48c22cb863d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:18.753362 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9efde03f-a94d-4073-8572-d48c22cb863d service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:18.772348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 954/1930] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:26:18 2022] GET /identity/v3/auth/tokens => generated 3349 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:26:18.998057 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8e5215-5980-45d1-ab92-495b5ee1a96a None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:19.008575 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:19.068036 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8e5215-5980-45d1-ab92-495b5ee1a96a None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:19.072439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a74007-9550-4d92-80a1-bba794201390 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:19.073071 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a74007-9550-4d92-80a1-bba794201390 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:19.073357 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a74007-9550-4d92-80a1-bba794201390 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:19.073624 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a74007-9550-4d92-80a1-bba794201390 admin admin] PATH_INFO: `/v3/users/4f619ab4b6e94d23bd883ef3b4b96c14` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:19.074747 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a74007-9550-4d92-80a1-bba794201390 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4f619ab4b6e94d23bd883ef3b4b96c14)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:19.076647 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a74007-9550-4d92-80a1-bba794201390 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:19.123789 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-54a74007-9550-4d92-80a1-bba794201390 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4f619ab4b6e94d23bd883ef3b4b96c14'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:19.131757 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-54a74007-9550-4d92-80a1-bba794201390 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4f619ab4b6e94d23bd883ef3b4b96c14'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:19.148494 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-54a74007-9550-4d92-80a1-bba794201390 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4f619ab4b6e94d23bd883ef3b4b96c14'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:19.149345 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-54a74007-9550-4d92-80a1-bba794201390 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4f619ab4b6e94d23bd883ef3b4b96c14'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:19.149949 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-54a74007-9550-4d92-80a1-bba794201390 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4f619ab4b6e94d23bd883ef3b4b96c14'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:19.166140 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 977/1931] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:26:18 2022] DELETE /identity/v3/users/4f619ab4b6e94d23bd883ef3b4b96c14 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:26:19.171015 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459aba5b-1c6f-4777-93f6-8e4d1431be31 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:19.171530 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459aba5b-1c6f-4777-93f6-8e4d1431be31 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:19.171530 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459aba5b-1c6f-4777-93f6-8e4d1431be31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:19.193367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-459aba5b-1c6f-4777-93f6-8e4d1431be31 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:19.194587 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-459aba5b-1c6f-4777-93f6-8e4d1431be31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:19.256086 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 955/1932] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:19.266196 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a74007-9550-4d92-80a1-bba794201390 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:19.268074 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-54a74007-9550-4d92-80a1-bba794201390 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:19.328014 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a74007-9550-4d92-80a1-bba794201390 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:19.333718 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea0fa04-488d-4ac1-882e-421ac5eab936 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:19.334486 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea0fa04-488d-4ac1-882e-421ac5eab936 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:19.334762 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea0fa04-488d-4ac1-882e-421ac5eab936 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:19.335033 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea0fa04-488d-4ac1-882e-421ac5eab936 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:19.344682 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ea0fa04-488d-4ac1-882e-421ac5eab936 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:19.346287 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ea0fa04-488d-4ac1-882e-421ac5eab936 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:19.361473 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 978/1933] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:26:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:26:19.794489 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459aba5b-1c6f-4777-93f6-8e4d1431be31 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:19.796088 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-459aba5b-1c6f-4777-93f6-8e4d1431be31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:19.809445 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459aba5b-1c6f-4777-93f6-8e4d1431be31 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:19.826630 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64e893a3-d1e7-4997-a284-702189907795 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:19.827285 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e893a3-d1e7-4997-a284-702189907795 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:19.827557 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e893a3-d1e7-4997-a284-702189907795 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:19.827987 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e893a3-d1e7-4997-a284-702189907795 admin admin] PATH_INFO: `/v3/projects/23904d82da0244c0aaa0523d563e2c90` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:19.829168 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64e893a3-d1e7-4997-a284-702189907795 admin admin] RBAC: Authorizing `identity:delete_project(project_id=23904d82da0244c0aaa0523d563e2c90)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:19.832557 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64e893a3-d1e7-4997-a284-702189907795 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:19.899041 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-64e893a3-d1e7-4997-a284-702189907795 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '23904d82da0244c0aaa0523d563e2c90', 'request_id': 'req-64e893a3-d1e7-4997-a284-702189907795'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:19.908260 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-64e893a3-d1e7-4997-a284-702189907795 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '23904d82da0244c0aaa0523d563e2c90', 'request_id': 'req-64e893a3-d1e7-4997-a284-702189907795'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:19.914967 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-64e893a3-d1e7-4997-a284-702189907795 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '23904d82da0244c0aaa0523d563e2c90', 'request_id': 'req-64e893a3-d1e7-4997-a284-702189907795'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:19.927893 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:19.932907 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-64e893a3-d1e7-4997-a284-702189907795 admin admin] The token cache is being invalidate because project 23904d82da0244c0aaa0523d563e2c90 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:26:19.935729 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-64e893a3-d1e7-4997-a284-702189907795 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:19.937409 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-64e893a3-d1e7-4997-a284-702189907795 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:19.943280 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 956/1934] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:26:19 2022] DELETE /identity/v3/projects/23904d82da0244c0aaa0523d563e2c90 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:19.947476 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea0fa04-488d-4ac1-882e-421ac5eab936 service neutron] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:19.962400 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea0fa04-488d-4ac1-882e-421ac5eab936 service neutron] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:20.009752 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:20.062588 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b771731c-8d37-4a7d-b7ba-0c6b8fbfc7c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:20.062588 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b771731c-8d37-4a7d-b7ba-0c6b8fbfc7c5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:20.062588 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b771731c-8d37-4a7d-b7ba-0c6b8fbfc7c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:20.062588 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b771731c-8d37-4a7d-b7ba-0c6b8fbfc7c5 admin admin] PATH_INFO: `/v3/users/0e6e509a188849ed8ed3d92911588d35` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:20.062972 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b771731c-8d37-4a7d-b7ba-0c6b8fbfc7c5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0e6e509a188849ed8ed3d92911588d35)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:20.066484 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b771731c-8d37-4a7d-b7ba-0c6b8fbfc7c5 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:20.115968 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-b771731c-8d37-4a7d-b7ba-0c6b8fbfc7c5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0e6e509a188849ed8ed3d92911588d35'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:20.123939 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-b771731c-8d37-4a7d-b7ba-0c6b8fbfc7c5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0e6e509a188849ed8ed3d92911588d35'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:20.136828 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-b771731c-8d37-4a7d-b7ba-0c6b8fbfc7c5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0e6e509a188849ed8ed3d92911588d35'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:20.137529 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-b771731c-8d37-4a7d-b7ba-0c6b8fbfc7c5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0e6e509a188849ed8ed3d92911588d35'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:20.138135 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-b771731c-8d37-4a7d-b7ba-0c6b8fbfc7c5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0e6e509a188849ed8ed3d92911588d35'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:20.154656 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 979/1935] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:26:19 2022] DELETE /identity/v3/users/0e6e509a188849ed8ed3d92911588d35 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:20.441268 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64e893a3-d1e7-4997-a284-702189907795 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:20.443640 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-64e893a3-d1e7-4997-a284-702189907795 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:20.460398 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64e893a3-d1e7-4997-a284-702189907795 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:20.518843 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26d83a0-6915-42b4-b640-90edaed3e7a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:20.519566 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26d83a0-6915-42b4-b640-90edaed3e7a2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:20.519850 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26d83a0-6915-42b4-b640-90edaed3e7a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:20.520131 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26d83a0-6915-42b4-b640-90edaed3e7a2 admin admin] PATH_INFO: `/v3/projects/a68e8a327fd34d7a84dcf1d73112a917` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:20.521345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26d83a0-6915-42b4-b640-90edaed3e7a2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a68e8a327fd34d7a84dcf1d73112a917)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:20.523253 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26d83a0-6915-42b4-b640-90edaed3e7a2 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:20.569360 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f26d83a0-6915-42b4-b640-90edaed3e7a2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a68e8a327fd34d7a84dcf1d73112a917', 'request_id': 'req-f26d83a0-6915-42b4-b640-90edaed3e7a2'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:20.576739 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f26d83a0-6915-42b4-b640-90edaed3e7a2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a68e8a327fd34d7a84dcf1d73112a917', 'request_id': 'req-f26d83a0-6915-42b4-b640-90edaed3e7a2'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:20.582832 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f26d83a0-6915-42b4-b640-90edaed3e7a2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a68e8a327fd34d7a84dcf1d73112a917', 'request_id': 'req-f26d83a0-6915-42b4-b640-90edaed3e7a2'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:20.593623 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f26d83a0-6915-42b4-b640-90edaed3e7a2 admin admin] The token cache is being invalidate because project a68e8a327fd34d7a84dcf1d73112a917 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:26:20.593912 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f26d83a0-6915-42b4-b640-90edaed3e7a2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:20.594543 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f26d83a0-6915-42b4-b640-90edaed3e7a2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:20.596026 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 957/1936] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:26:20 2022] DELETE /identity/v3/projects/a68e8a327fd34d7a84dcf1d73112a917 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:20.601469 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b8c209-dee6-4057-b03b-f5e0d26396c9 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:20.601755 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b8c209-dee6-4057-b03b-f5e0d26396c9 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:20.602029 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b8c209-dee6-4057-b03b-f5e0d26396c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:20.621178 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-f7b8c209-dee6-4057-b03b-f5e0d26396c9 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:20.622391 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-f7b8c209-dee6-4057-b03b-f5e0d26396c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:20.688392 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 980/1937] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:20.696439 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78bc89d-2b82-4f7f-a99a-727bc3d04b40 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:20.696719 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78bc89d-2b82-4f7f-a99a-727bc3d04b40 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:20.696990 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78bc89d-2b82-4f7f-a99a-727bc3d04b40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:20.716960 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-f78bc89d-2b82-4f7f-a99a-727bc3d04b40 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:20.718178 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f78bc89d-2b82-4f7f-a99a-727bc3d04b40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:20.726535 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 958/1938] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:20.730790 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b8c209-dee6-4057-b03b-f5e0d26396c9 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:20.740594 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b8c209-dee6-4057-b03b-f5e0d26396c9 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:20.745231 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c526ecb-1b9a-4a0e-9c1b-2517dfa86b4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:20.745825 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c526ecb-1b9a-4a0e-9c1b-2517dfa86b4e admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:20.746106 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c526ecb-1b9a-4a0e-9c1b-2517dfa86b4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:20.746378 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c526ecb-1b9a-4a0e-9c1b-2517dfa86b4e admin admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:20.747019 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c526ecb-1b9a-4a0e-9c1b-2517dfa86b4e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:26:20.747320 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c526ecb-1b9a-4a0e-9c1b-2517dfa86b4e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:20.748918 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c526ecb-1b9a-4a0e-9c1b-2517dfa86b4e admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:20.760478 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 981/1939] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:26:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:26:20.764658 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78bc89d-2b82-4f7f-a99a-727bc3d04b40 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:20.777105 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78bc89d-2b82-4f7f-a99a-727bc3d04b40 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:20.782357 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa2ee35-1b86-4e0e-a848-4f9a18042513 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:20.782952 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa2ee35-1b86-4e0e-a848-4f9a18042513 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:20.783227 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa2ee35-1b86-4e0e-a848-4f9a18042513 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:20.783493 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa2ee35-1b86-4e0e-a848-4f9a18042513 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:20.784770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa2ee35-1b86-4e0e-a848-4f9a18042513 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:20.786689 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa2ee35-1b86-4e0e-a848-4f9a18042513 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:20.819845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 959/1940] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:26:20 2022] POST /identity/v3/projects => generated 365 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:26:20.821875 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c526ecb-1b9a-4a0e-9c1b-2517dfa86b4e admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:20.835180 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c526ecb-1b9a-4a0e-9c1b-2517dfa86b4e admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:20.880905 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:20.920618 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96125358-7fbe-4780-9f81-3dd959182901 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:20.921268 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96125358-7fbe-4780-9f81-3dd959182901 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:20.921634 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96125358-7fbe-4780-9f81-3dd959182901 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:20.921996 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96125358-7fbe-4780-9f81-3dd959182901 admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:20.922689 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96125358-7fbe-4780-9f81-3dd959182901 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:20.924610 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96125358-7fbe-4780-9f81-3dd959182901 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:20.955009 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 982/1941] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:26:20 2022] POST /identity/v3/users => generated 283 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:26:20.957625 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa2ee35-1b86-4e0e-a848-4f9a18042513 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:20.968332 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa2ee35-1b86-4e0e-a848-4f9a18042513 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:20.974387 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38daff8f-4948-4036-9bf0-d9974bc84295 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:20.975350 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38daff8f-4948-4036-9bf0-d9974bc84295 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:20.975625 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38daff8f-4948-4036-9bf0-d9974bc84295 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:20.975892 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38daff8f-4948-4036-9bf0-d9974bc84295 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:20.976545 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38daff8f-4948-4036-9bf0-d9974bc84295 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:20.978176 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38daff8f-4948-4036-9bf0-d9974bc84295 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:20.992299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 960/1942] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:26:20 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:26:20.996556 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96125358-7fbe-4780-9f81-3dd959182901 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:21.007625 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96125358-7fbe-4780-9f81-3dd959182901 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:21.013417 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15edc32-621b-4286-86ab-3a041c045f8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:21.014026 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15edc32-621b-4286-86ab-3a041c045f8d admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:21.014295 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15edc32-621b-4286-86ab-3a041c045f8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:21.014564 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15edc32-621b-4286-86ab-3a041c045f8d admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:21.015208 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c15edc32-621b-4286-86ab-3a041c045f8d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:21.016878 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c15edc32-621b-4286-86ab-3a041c045f8d admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:21.028717 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 983/1943] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:26:20 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:26:21.033077 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38daff8f-4948-4036-9bf0-d9974bc84295 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:21.045168 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38daff8f-4948-4036-9bf0-d9974bc84295 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:21.051790 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd078db8-f2af-4210-ae08-d9413199cb55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:21.053242 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd078db8-f2af-4210-ae08-d9413199cb55 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:21.053524 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd078db8-f2af-4210-ae08-d9413199cb55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:21.053904 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd078db8-f2af-4210-ae08-d9413199cb55 admin admin] PATH_INFO: `/v3/projects/c16b71d2277541d795b9c62c2b511efb/users/fabdc4c6696142418c89b2d3d18f80ad/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:21.076034 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd078db8-f2af-4210-ae08-d9413199cb55 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c16b71d2277541d795b9c62c2b511efb, user_id=fabdc4c6696142418c89b2d3d18f80ad, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:21.080230 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd078db8-f2af-4210-ae08-d9413199cb55 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:21.097916 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 961/1944] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:26:21 2022] PUT /identity/v3/projects/c16b71d2277541d795b9c62c2b511efb/users/fabdc4c6696142418c89b2d3d18f80ad/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:21.109869 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840bb9c8-8e25-47af-895f-22b0e0f547d9 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:21.110153 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840bb9c8-8e25-47af-895f-22b0e0f547d9 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:21.110422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840bb9c8-8e25-47af-895f-22b0e0f547d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:21.140552 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-840bb9c8-8e25-47af-895f-22b0e0f547d9 None None] MFA Rules not processed for user `fabdc4c6696142418c89b2d3d18f80ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:21.141815 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-840bb9c8-8e25-47af-895f-22b0e0f547d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:21.221779 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 984/1945] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:21 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:21.226215 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd078db8-f2af-4210-ae08-d9413199cb55 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:21.258101 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:21.293164 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd078db8-f2af-4210-ae08-d9413199cb55 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:21.298863 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474dd659-50fa-4104-8724-9ecf24ca4892 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:21.298863 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474dd659-50fa-4104-8724-9ecf24ca4892 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:21.298863 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474dd659-50fa-4104-8724-9ecf24ca4892 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:21.301233 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474dd659-50fa-4104-8724-9ecf24ca4892 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:21.301378 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-474dd659-50fa-4104-8724-9ecf24ca4892 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:21.303298 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-474dd659-50fa-4104-8724-9ecf24ca4892 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:21.333897 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 962/1946] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:26:21 2022] POST /identity/v3/projects => generated 365 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:26:21.338065 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840bb9c8-8e25-47af-895f-22b0e0f547d9 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:21.352227 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840bb9c8-8e25-47af-895f-22b0e0f547d9 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:21.421955 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1949d1-50f7-4eea-9ebe-9b6fba551c8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:21.422574 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1949d1-50f7-4eea-9ebe-9b6fba551c8e admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:21.423221 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1949d1-50f7-4eea-9ebe-9b6fba551c8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:21.423221 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1949d1-50f7-4eea-9ebe-9b6fba551c8e admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:21.423926 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1949d1-50f7-4eea-9ebe-9b6fba551c8e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:21.426910 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1949d1-50f7-4eea-9ebe-9b6fba551c8e admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:21.451134 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 985/1947] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:26:21 2022] POST /identity/v3/users => generated 289 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:26:21.459309 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474dd659-50fa-4104-8724-9ecf24ca4892 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:21.475343 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474dd659-50fa-4104-8724-9ecf24ca4892 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:21.479944 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6245985-1ce7-4c81-bfdb-9eb9ec559a7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:21.480551 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6245985-1ce7-4c81-bfdb-9eb9ec559a7b admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:21.480827 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6245985-1ce7-4c81-bfdb-9eb9ec559a7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:21.481137 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6245985-1ce7-4c81-bfdb-9eb9ec559a7b admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:21.481785 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6245985-1ce7-4c81-bfdb-9eb9ec559a7b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:21.483403 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6245985-1ce7-4c81-bfdb-9eb9ec559a7b admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:21.496595 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 963/1948] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:26:21 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:26:21.499707 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1949d1-50f7-4eea-9ebe-9b6fba551c8e admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:21.510535 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1949d1-50f7-4eea-9ebe-9b6fba551c8e admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:21.517805 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a2c9ad-a4d0-4b9f-a336-7f67f3565b7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:21.519163 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a2c9ad-a4d0-4b9f-a336-7f67f3565b7a admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:21.519441 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a2c9ad-a4d0-4b9f-a336-7f67f3565b7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:21.519714 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a2c9ad-a4d0-4b9f-a336-7f67f3565b7a admin admin] PATH_INFO: `/v3/projects/02bec56d7317465a9e31c537b71633e0/users/0383b823d3ae41ed953054c8fc09630e/roles/414286927fdd44688107ff5e10235acf` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:21.538749 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79a2c9ad-a4d0-4b9f-a336-7f67f3565b7a admin admin] RBAC: Authorizing `identity:create_grant(project_id=02bec56d7317465a9e31c537b71633e0, user_id=0383b823d3ae41ed953054c8fc09630e, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:21.541466 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79a2c9ad-a4d0-4b9f-a336-7f67f3565b7a admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:21.556205 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 986/1949] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:26:21 2022] PUT /identity/v3/projects/02bec56d7317465a9e31c537b71633e0/users/0383b823d3ae41ed953054c8fc09630e/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:26:21.560374 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6245985-1ce7-4c81-bfdb-9eb9ec559a7b admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:21.570977 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6245985-1ce7-4c81-bfdb-9eb9ec559a7b admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:21.631379 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d97804-6ef3-44ff-97a3-da298fa0e880 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:21.631999 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d97804-6ef3-44ff-97a3-da298fa0e880 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:21.632276 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d97804-6ef3-44ff-97a3-da298fa0e880 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:21.632547 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d97804-6ef3-44ff-97a3-da298fa0e880 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:21.633244 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d97804-6ef3-44ff-97a3-da298fa0e880 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:21.634893 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d97804-6ef3-44ff-97a3-da298fa0e880 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:21.645757 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 964/1950] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:26:21 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 27 01:26:21.650030 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a2c9ad-a4d0-4b9f-a336-7f67f3565b7a admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:21.660945 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a2c9ad-a4d0-4b9f-a336-7f67f3565b7a admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:21.665522 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606beae8-b9a4-476a-b7ca-e0ec33400cf9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:21.666160 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606beae8-b9a4-476a-b7ca-e0ec33400cf9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:21.666431 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606beae8-b9a4-476a-b7ca-e0ec33400cf9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:21.666704 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606beae8-b9a4-476a-b7ca-e0ec33400cf9 admin admin] PATH_INFO: `/v3/projects/02bec56d7317465a9e31c537b71633e0/users/0383b823d3ae41ed953054c8fc09630e/roles/414286927fdd44688107ff5e10235acf` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:21.668580 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606beae8-b9a4-476a-b7ca-e0ec33400cf9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=02bec56d7317465a9e31c537b71633e0, user_id=0383b823d3ae41ed953054c8fc09630e, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:21.671220 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606beae8-b9a4-476a-b7ca-e0ec33400cf9 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:21.674138 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:21.683707 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 987/1951] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:26:21 2022] PUT /identity/v3/projects/02bec56d7317465a9e31c537b71633e0/users/0383b823d3ae41ed953054c8fc09630e/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:21.694243 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570c68d0-9da2-4f3c-b677-4036889f06b7 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:21.694534 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570c68d0-9da2-4f3c-b677-4036889f06b7 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:21.694843 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570c68d0-9da2-4f3c-b677-4036889f06b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:21.728925 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-570c68d0-9da2-4f3c-b677-4036889f06b7 None None] MFA Rules not processed for user `0383b823d3ae41ed953054c8fc09630e`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:21.730156 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-570c68d0-9da2-4f3c-b677-4036889f06b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:21.805602 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 965/1952] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:21 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:21.820054 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606beae8-b9a4-476a-b7ca-e0ec33400cf9 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:21.823457 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-606beae8-b9a4-476a-b7ca-e0ec33400cf9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:21.882963 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606beae8-b9a4-476a-b7ca-e0ec33400cf9 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:21.886746 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75e8e9e-2992-4743-87b8-2363a1d35bd6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:21.887360 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75e8e9e-2992-4743-87b8-2363a1d35bd6 service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:21.887672 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75e8e9e-2992-4743-87b8-2363a1d35bd6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:21.887976 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75e8e9e-2992-4743-87b8-2363a1d35bd6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:21.899891 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a75e8e9e-2992-4743-87b8-2363a1d35bd6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:21.901355 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a75e8e9e-2992-4743-87b8-2363a1d35bd6 service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:21.911527 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 988/1953] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:26:21 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:26:22.357371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-570c68d0-9da2-4f3c-b677-4036889f06b7 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:22.383473 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:22.422606 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-570c68d0-9da2-4f3c-b677-4036889f06b7 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:22.427401 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c943618b-2e7f-4aa6-9bff-a7d6809bd7d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:22.428029 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c943618b-2e7f-4aa6-9bff-a7d6809bd7d1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:22.428313 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c943618b-2e7f-4aa6-9bff-a7d6809bd7d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:22.428583 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c943618b-2e7f-4aa6-9bff-a7d6809bd7d1 admin admin] PATH_INFO: `/v3/users/fabdc4c6696142418c89b2d3d18f80ad` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:22.429986 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c943618b-2e7f-4aa6-9bff-a7d6809bd7d1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fabdc4c6696142418c89b2d3d18f80ad)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:22.431910 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c943618b-2e7f-4aa6-9bff-a7d6809bd7d1 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:22.482107 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-c943618b-2e7f-4aa6-9bff-a7d6809bd7d1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fabdc4c6696142418c89b2d3d18f80ad'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:22.491252 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-c943618b-2e7f-4aa6-9bff-a7d6809bd7d1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fabdc4c6696142418c89b2d3d18f80ad'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:22.492152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-c943618b-2e7f-4aa6-9bff-a7d6809bd7d1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fabdc4c6696142418c89b2d3d18f80ad'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:22.505553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-c943618b-2e7f-4aa6-9bff-a7d6809bd7d1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fabdc4c6696142418c89b2d3d18f80ad'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:22.512301 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-c943618b-2e7f-4aa6-9bff-a7d6809bd7d1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fabdc4c6696142418c89b2d3d18f80ad'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:22.518774 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 966/1954] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:26:22 2022] DELETE /identity/v3/users/fabdc4c6696142418c89b2d3d18f80ad => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:22.523701 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133a999e-d5a2-49cd-97a9-53677eaa6fca None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:22.524147 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133a999e-d5a2-49cd-97a9-53677eaa6fca None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:22.524260 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133a999e-d5a2-49cd-97a9-53677eaa6fca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:22.544966 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-133a999e-d5a2-49cd-97a9-53677eaa6fca None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:22.546204 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-133a999e-d5a2-49cd-97a9-53677eaa6fca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:22.605213 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 989/1955] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:22.615288 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c943618b-2e7f-4aa6-9bff-a7d6809bd7d1 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:22.616703 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-c943618b-2e7f-4aa6-9bff-a7d6809bd7d1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:22.673480 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c943618b-2e7f-4aa6-9bff-a7d6809bd7d1 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:22.677888 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2bb61c-db6e-4c67-a1bb-ae1854ab43ed service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:22.678485 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2bb61c-db6e-4c67-a1bb-ae1854ab43ed service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:22.678762 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2bb61c-db6e-4c67-a1bb-ae1854ab43ed service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:22.679030 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2bb61c-db6e-4c67-a1bb-ae1854ab43ed service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:22.688137 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2bb61c-db6e-4c67-a1bb-ae1854ab43ed service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:22.689601 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2bb61c-db6e-4c67-a1bb-ae1854ab43ed service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:22.699566 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 967/1956] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:26:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:26:22.965926 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133a999e-d5a2-49cd-97a9-53677eaa6fca None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:22.967638 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-133a999e-d5a2-49cd-97a9-53677eaa6fca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:22.981120 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133a999e-d5a2-49cd-97a9-53677eaa6fca None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:22.993739 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e813c1-1fa0-4dda-8aed-1928707c45c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:22.994374 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e813c1-1fa0-4dda-8aed-1928707c45c6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:22.994646 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e813c1-1fa0-4dda-8aed-1928707c45c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:22.994910 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e813c1-1fa0-4dda-8aed-1928707c45c6 admin admin] PATH_INFO: `/v3/projects/c16b71d2277541d795b9c62c2b511efb` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:22.996073 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e813c1-1fa0-4dda-8aed-1928707c45c6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c16b71d2277541d795b9c62c2b511efb)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:22.998067 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e813c1-1fa0-4dda-8aed-1928707c45c6 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:23.046200 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-62e813c1-1fa0-4dda-8aed-1928707c45c6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c16b71d2277541d795b9c62c2b511efb', 'request_id': 'req-62e813c1-1fa0-4dda-8aed-1928707c45c6'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:23.054057 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-62e813c1-1fa0-4dda-8aed-1928707c45c6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c16b71d2277541d795b9c62c2b511efb', 'request_id': 'req-62e813c1-1fa0-4dda-8aed-1928707c45c6'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:23.060601 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-62e813c1-1fa0-4dda-8aed-1928707c45c6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c16b71d2277541d795b9c62c2b511efb', 'request_id': 'req-62e813c1-1fa0-4dda-8aed-1928707c45c6'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:23.072094 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-62e813c1-1fa0-4dda-8aed-1928707c45c6 admin admin] The token cache is being invalidate because project c16b71d2277541d795b9c62c2b511efb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:26:23.072501 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-62e813c1-1fa0-4dda-8aed-1928707c45c6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:23.073178 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-62e813c1-1fa0-4dda-8aed-1928707c45c6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:23.074773 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 990/1957] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:26:22 2022] DELETE /identity/v3/projects/c16b71d2277541d795b9c62c2b511efb => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:23.078824 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2bb61c-db6e-4c67-a1bb-ae1854ab43ed service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:23.088872 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2bb61c-db6e-4c67-a1bb-ae1854ab43ed service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:23.130210 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:23.146159 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-588dd4ba-6cd0-44da-8f1f-33c2acbd1bb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:23.146722 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588dd4ba-6cd0-44da-8f1f-33c2acbd1bb3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:23.147000 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588dd4ba-6cd0-44da-8f1f-33c2acbd1bb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:23.147272 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588dd4ba-6cd0-44da-8f1f-33c2acbd1bb3 admin admin] PATH_INFO: `/v3/users/0383b823d3ae41ed953054c8fc09630e` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:23.148375 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-588dd4ba-6cd0-44da-8f1f-33c2acbd1bb3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0383b823d3ae41ed953054c8fc09630e)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:23.150357 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-588dd4ba-6cd0-44da-8f1f-33c2acbd1bb3 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:23.196472 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-588dd4ba-6cd0-44da-8f1f-33c2acbd1bb3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0383b823d3ae41ed953054c8fc09630e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:23.204119 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-588dd4ba-6cd0-44da-8f1f-33c2acbd1bb3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0383b823d3ae41ed953054c8fc09630e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:23.204727 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-588dd4ba-6cd0-44da-8f1f-33c2acbd1bb3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0383b823d3ae41ed953054c8fc09630e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:23.217167 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-588dd4ba-6cd0-44da-8f1f-33c2acbd1bb3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0383b823d3ae41ed953054c8fc09630e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:23.224937 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-588dd4ba-6cd0-44da-8f1f-33c2acbd1bb3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0383b823d3ae41ed953054c8fc09630e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:23.230825 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 968/1958] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:26:23 2022] DELETE /identity/v3/users/0383b823d3ae41ed953054c8fc09630e => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:23.698338 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e813c1-1fa0-4dda-8aed-1928707c45c6 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:23.699747 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-62e813c1-1fa0-4dda-8aed-1928707c45c6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:23.714764 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e813c1-1fa0-4dda-8aed-1928707c45c6 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:23.735649 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:23.776465 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2860bcb6-b665-423a-8545-4f36eb91b705 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:23.777116 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2860bcb6-b665-423a-8545-4f36eb91b705 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:23.777505 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2860bcb6-b665-423a-8545-4f36eb91b705 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:23.777778 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2860bcb6-b665-423a-8545-4f36eb91b705 admin admin] PATH_INFO: `/v3/projects/02bec56d7317465a9e31c537b71633e0` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:23.778890 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2860bcb6-b665-423a-8545-4f36eb91b705 admin admin] RBAC: Authorizing `identity:delete_project(project_id=02bec56d7317465a9e31c537b71633e0)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:23.780785 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2860bcb6-b665-423a-8545-4f36eb91b705 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:23.827573 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-2860bcb6-b665-423a-8545-4f36eb91b705 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '02bec56d7317465a9e31c537b71633e0', 'request_id': 'req-2860bcb6-b665-423a-8545-4f36eb91b705'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:23.836316 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-2860bcb6-b665-423a-8545-4f36eb91b705 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '02bec56d7317465a9e31c537b71633e0', 'request_id': 'req-2860bcb6-b665-423a-8545-4f36eb91b705'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:23.842571 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-2860bcb6-b665-423a-8545-4f36eb91b705 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '02bec56d7317465a9e31c537b71633e0', 'request_id': 'req-2860bcb6-b665-423a-8545-4f36eb91b705'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:23.853115 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-2860bcb6-b665-423a-8545-4f36eb91b705 admin admin] The token cache is being invalidate because project 02bec56d7317465a9e31c537b71633e0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:26:23.853399 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-2860bcb6-b665-423a-8545-4f36eb91b705 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:23.854023 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-2860bcb6-b665-423a-8545-4f36eb91b705 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:23.855498 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 991/1959] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:26:23 2022] DELETE /identity/v3/projects/02bec56d7317465a9e31c537b71633e0 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:23.860569 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6d952d-6684-430c-bfc8-b866b7e41d71 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:23.861023 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6d952d-6684-430c-bfc8-b866b7e41d71 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:23.861161 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6d952d-6684-430c-bfc8-b866b7e41d71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:23.882258 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-7a6d952d-6684-430c-bfc8-b866b7e41d71 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:23.883447 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-7a6d952d-6684-430c-bfc8-b866b7e41d71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:23.944273 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 969/1960] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:23.953903 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1933a1-12bd-471c-962c-56465cf7b099 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:23.954186 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1933a1-12bd-471c-962c-56465cf7b099 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:23.954481 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1933a1-12bd-471c-962c-56465cf7b099 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:23.974622 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-5a1933a1-12bd-471c-962c-56465cf7b099 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:23.975804 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-5a1933a1-12bd-471c-962c-56465cf7b099 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:23.983688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 992/1961] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:23.987957 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a6d952d-6684-430c-bfc8-b866b7e41d71 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:23.997640 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a6d952d-6684-430c-bfc8-b866b7e41d71 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:24.002218 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1c41a1-d8e5-4b6c-a272-63f816e46e3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:24.003308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1c41a1-d8e5-4b6c-a272-63f816e46e3c admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:24.003586 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1c41a1-d8e5-4b6c-a272-63f816e46e3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:24.003856 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1c41a1-d8e5-4b6c-a272-63f816e46e3c admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:24.004499 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c1c41a1-d8e5-4b6c-a272-63f816e46e3c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:26:24.004797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c1c41a1-d8e5-4b6c-a272-63f816e46e3c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:24.006453 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c1c41a1-d8e5-4b6c-a272-63f816e46e3c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:24.030492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 970/1962] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:26:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:26:24.031587 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1933a1-12bd-471c-962c-56465cf7b099 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:24.041879 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1933a1-12bd-471c-962c-56465cf7b099 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:24.046468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddbd85fe-3bb1-4b0f-a19e-f2edf9b4c503 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:24.047064 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbd85fe-3bb1-4b0f-a19e-f2edf9b4c503 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:24.047339 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbd85fe-3bb1-4b0f-a19e-f2edf9b4c503 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:24.047604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbd85fe-3bb1-4b0f-a19e-f2edf9b4c503 admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:24.048297 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddbd85fe-3bb1-4b0f-a19e-f2edf9b4c503 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:24.050348 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddbd85fe-3bb1-4b0f-a19e-f2edf9b4c503 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:24.078227 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 993/1963] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:26:24 2022] POST /identity/v3/projects => generated 355 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:26:24.083004 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1c41a1-d8e5-4b6c-a272-63f816e46e3c admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:24.093038 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1c41a1-d8e5-4b6c-a272-63f816e46e3c admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:24.140116 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:24.151373 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b276692a-380b-4278-8ed1-2d72cc30e61a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:24.151978 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b276692a-380b-4278-8ed1-2d72cc30e61a admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:24.152259 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b276692a-380b-4278-8ed1-2d72cc30e61a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:24.152534 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b276692a-380b-4278-8ed1-2d72cc30e61a admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:24.153262 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b276692a-380b-4278-8ed1-2d72cc30e61a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:24.155315 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b276692a-380b-4278-8ed1-2d72cc30e61a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:24.176955 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 971/1964] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:26:24 2022] POST /identity/v3/users => generated 278 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:26:24.181428 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddbd85fe-3bb1-4b0f-a19e-f2edf9b4c503 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:24.192432 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddbd85fe-3bb1-4b0f-a19e-f2edf9b4c503 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:24.197251 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d47e43-29a3-40d8-ab5b-2018aba9ca14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:24.197815 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d47e43-29a3-40d8-ab5b-2018aba9ca14 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:24.198091 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d47e43-29a3-40d8-ab5b-2018aba9ca14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:24.198363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d47e43-29a3-40d8-ab5b-2018aba9ca14 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:24.199021 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6d47e43-29a3-40d8-ab5b-2018aba9ca14 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:24.200847 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6d47e43-29a3-40d8-ab5b-2018aba9ca14 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:24.211239 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 994/1965] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:26:24 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:26:24.215122 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b276692a-380b-4278-8ed1-2d72cc30e61a admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:24.224989 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b276692a-380b-4278-8ed1-2d72cc30e61a admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:24.229626 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4753f9a1-2cca-4fe1-a1e9-e148aaaa9155 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:24.230213 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4753f9a1-2cca-4fe1-a1e9-e148aaaa9155 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:24.230494 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4753f9a1-2cca-4fe1-a1e9-e148aaaa9155 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:24.230763 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4753f9a1-2cca-4fe1-a1e9-e148aaaa9155 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:24.231403 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4753f9a1-2cca-4fe1-a1e9-e148aaaa9155 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:24.232984 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4753f9a1-2cca-4fe1-a1e9-e148aaaa9155 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:24.243995 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 972/1966] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:26:24 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 27 01:26:24.248106 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d47e43-29a3-40d8-ab5b-2018aba9ca14 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:24.258602 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d47e43-29a3-40d8-ab5b-2018aba9ca14 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:24.263380 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66325224-544d-4d95-bc7f-0d656559c8f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:24.264060 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66325224-544d-4d95-bc7f-0d656559c8f7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:24.264362 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66325224-544d-4d95-bc7f-0d656559c8f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:24.264635 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66325224-544d-4d95-bc7f-0d656559c8f7 admin admin] PATH_INFO: `/v3/projects/9879b9937ead4ef3a6c1c9de77d3a63d/users/f4da0cda7fbd4ede8d3051a4544c2e8c/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:24.279577 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66325224-544d-4d95-bc7f-0d656559c8f7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9879b9937ead4ef3a6c1c9de77d3a63d, user_id=f4da0cda7fbd4ede8d3051a4544c2e8c, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:24.282318 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66325224-544d-4d95-bc7f-0d656559c8f7 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:24.294362 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 995/1967] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:26:24 2022] PUT /identity/v3/projects/9879b9937ead4ef3a6c1c9de77d3a63d/users/f4da0cda7fbd4ede8d3051a4544c2e8c/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:24.305962 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752e9de7-cc93-4986-a8aa-ad1698579031 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:24.306244 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752e9de7-cc93-4986-a8aa-ad1698579031 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:24.306544 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752e9de7-cc93-4986-a8aa-ad1698579031 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:24.334327 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-752e9de7-cc93-4986-a8aa-ad1698579031 None None] MFA Rules not processed for user `f4da0cda7fbd4ede8d3051a4544c2e8c`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:24.335530 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-752e9de7-cc93-4986-a8aa-ad1698579031 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:24.403707 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 973/1968] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:24 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:24.407889 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66325224-544d-4d95-bc7f-0d656559c8f7 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:24.472746 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:24.498314 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66325224-544d-4d95-bc7f-0d656559c8f7 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:24.503583 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f340826c-f910-495f-8212-44b988e0da79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:24.504199 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f340826c-f910-495f-8212-44b988e0da79 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:24.504474 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f340826c-f910-495f-8212-44b988e0da79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:24.504741 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f340826c-f910-495f-8212-44b988e0da79 admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:24.505597 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f340826c-f910-495f-8212-44b988e0da79 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:24.507506 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f340826c-f910-495f-8212-44b988e0da79 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:24.569749 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 996/1969] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:26:24 2022] POST /identity/v3/projects => generated 357 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:26:24.572174 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752e9de7-cc93-4986-a8aa-ad1698579031 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:24.589031 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752e9de7-cc93-4986-a8aa-ad1698579031 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:24.670757 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f340826c-f910-495f-8212-44b988e0da79 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:24.672565 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-f340826c-f910-495f-8212-44b988e0da79 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:24.686579 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d248d5ca-b43a-4ee7-9640-12fac1dab882 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:24.687391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d248d5ca-b43a-4ee7-9640-12fac1dab882 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:24.687756 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d248d5ca-b43a-4ee7-9640-12fac1dab882 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:24.688150 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d248d5ca-b43a-4ee7-9640-12fac1dab882 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:24.688850 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d248d5ca-b43a-4ee7-9640-12fac1dab882 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:24.692387 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d248d5ca-b43a-4ee7-9640-12fac1dab882 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:24.717908 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 974/1970] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:26:24 2022] POST /identity/v3/users => generated 285 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:26:24.721669 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d248d5ca-b43a-4ee7-9640-12fac1dab882 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:24.750964 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d248d5ca-b43a-4ee7-9640-12fac1dab882 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:24.759622 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3df20c6-d409-48d5-a43b-6a67671f3481 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:24.760463 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3df20c6-d409-48d5-a43b-6a67671f3481 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:24.760807 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3df20c6-d409-48d5-a43b-6a67671f3481 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:24.761173 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3df20c6-d409-48d5-a43b-6a67671f3481 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:24.761892 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3df20c6-d409-48d5-a43b-6a67671f3481 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:24.765489 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3df20c6-d409-48d5-a43b-6a67671f3481 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:24.768147 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f340826c-f910-495f-8212-44b988e0da79 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:24.775310 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd09ba01-7c8b-447d-94c6-0b9d9f359c6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:24.775961 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd09ba01-7c8b-447d-94c6-0b9d9f359c6b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:24.776238 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd09ba01-7c8b-447d-94c6-0b9d9f359c6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:24.776506 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd09ba01-7c8b-447d-94c6-0b9d9f359c6b admin admin] PATH_INFO: `/v3/users/3768beee62fe403489672b08c0113239` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:24.777705 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd09ba01-7c8b-447d-94c6-0b9d9f359c6b admin admin] RBAC: Authorizing `identity:delete_user(user_id=3768beee62fe403489672b08c0113239)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:24.779632 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd09ba01-7c8b-447d-94c6-0b9d9f359c6b admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:24.787940 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 975/1971] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:26:24 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 27 01:26:24.791569 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3df20c6-d409-48d5-a43b-6a67671f3481 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:24.814687 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3df20c6-d409-48d5-a43b-6a67671f3481 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:24.827608 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4525af20-2a45-470c-8d50-5a596b7f40e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:24.828315 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4525af20-2a45-470c-8d50-5a596b7f40e8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:24.828594 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4525af20-2a45-470c-8d50-5a596b7f40e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:24.828875 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4525af20-2a45-470c-8d50-5a596b7f40e8 admin admin] PATH_INFO: `/v3/projects/d1b95477ea8e456bafe9470310b4ce3b/users/917d19d1f44c42969e0efccd0f11019a/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:24.849490 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4525af20-2a45-470c-8d50-5a596b7f40e8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d1b95477ea8e456bafe9470310b4ce3b, user_id=917d19d1f44c42969e0efccd0f11019a, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:24.852843 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4525af20-2a45-470c-8d50-5a596b7f40e8 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:24.856837 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-bd09ba01-7c8b-447d-94c6-0b9d9f359c6b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3768beee62fe403489672b08c0113239'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:24.872402 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 976/1972] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:26:24 2022] PUT /identity/v3/projects/d1b95477ea8e456bafe9470310b4ce3b/users/917d19d1f44c42969e0efccd0f11019a/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 82 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:24.875630 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-bd09ba01-7c8b-447d-94c6-0b9d9f359c6b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3768beee62fe403489672b08c0113239'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:24.877337 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4525af20-2a45-470c-8d50-5a596b7f40e8 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:24.898501 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-bd09ba01-7c8b-447d-94c6-0b9d9f359c6b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3768beee62fe403489672b08c0113239'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:24.904918 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-bd09ba01-7c8b-447d-94c6-0b9d9f359c6b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3768beee62fe403489672b08c0113239'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:24.905093 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:24.905702 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-bd09ba01-7c8b-447d-94c6-0b9d9f359c6b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3768beee62fe403489672b08c0113239'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:24.920152 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 997/1973] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:26:24 2022] DELETE /identity/v3/users/3768beee62fe403489672b08c0113239 => generated 0 bytes in 250 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:24.925113 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4f8e0b-0ac8-40ec-847b-b3fe5a6e7038 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:24.925448 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4f8e0b-0ac8-40ec-847b-b3fe5a6e7038 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:24.925725 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4f8e0b-0ac8-40ec-847b-b3fe5a6e7038 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:24.945105 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-6e4f8e0b-0ac8-40ec-847b-b3fe5a6e7038 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:24.946323 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-6e4f8e0b-0ac8-40ec-847b-b3fe5a6e7038 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:24.948986 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4525af20-2a45-470c-8d50-5a596b7f40e8 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:24.995747 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:25.017050 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 998/1974] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:26:25.020145 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1a8af4-8942-4905-8fbe-a8c606071ac5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:25.021619 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1a8af4-8942-4905-8fbe-a8c606071ac5 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:25.021886 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1a8af4-8942-4905-8fbe-a8c606071ac5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:25.022209 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1a8af4-8942-4905-8fbe-a8c606071ac5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:25.022896 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1a8af4-8942-4905-8fbe-a8c606071ac5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:25.024569 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1a8af4-8942-4905-8fbe-a8c606071ac5 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:25.030215 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e4f8e0b-0ac8-40ec-847b-b3fe5a6e7038 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:25.031756 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-6e4f8e0b-0ac8-40ec-847b-b3fe5a6e7038 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:25.037416 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 977/1975] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:26:24 2022] GET /identity/v3/roles => generated 1420 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:26:25.042041 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1a8af4-8942-4905-8fbe-a8c606071ac5 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:25.043737 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ca1a8af4-8942-4905-8fbe-a8c606071ac5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:25.057684 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1a8af4-8942-4905-8fbe-a8c606071ac5 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:25.062530 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62cc2cf8-da64-4fc9-8be1-ac77367fee80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:25.063207 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cc2cf8-da64-4fc9-8be1-ac77367fee80 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:25.063452 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cc2cf8-da64-4fc9-8be1-ac77367fee80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:25.063728 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cc2cf8-da64-4fc9-8be1-ac77367fee80 admin admin] PATH_INFO: `/v3/projects/d1b95477ea8e456bafe9470310b4ce3b/users/917d19d1f44c42969e0efccd0f11019a/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:25.066029 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62cc2cf8-da64-4fc9-8be1-ac77367fee80 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d1b95477ea8e456bafe9470310b4ce3b, user_id=917d19d1f44c42969e0efccd0f11019a, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:25.068812 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62cc2cf8-da64-4fc9-8be1-ac77367fee80 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:25.081253 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 978/1976] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:26:25 2022] PUT /identity/v3/projects/d1b95477ea8e456bafe9470310b4ce3b/users/917d19d1f44c42969e0efccd0f11019a/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:25.091606 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777af9c4-e61f-46a0-9dce-7848b51a0fe0 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:25.091882 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777af9c4-e61f-46a0-9dce-7848b51a0fe0 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:25.092159 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777af9c4-e61f-46a0-9dce-7848b51a0fe0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:25.099751 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e4f8e0b-0ac8-40ec-847b-b3fe5a6e7038 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:25.124582 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-777af9c4-e61f-46a0-9dce-7848b51a0fe0 None None] MFA Rules not processed for user `917d19d1f44c42969e0efccd0f11019a`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:25.125901 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-777af9c4-e61f-46a0-9dce-7848b51a0fe0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:25.168727 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b272bdd-161a-47e3-be35-e8f931739aef service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:25.169363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b272bdd-161a-47e3-be35-e8f931739aef service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:25.169636 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b272bdd-161a-47e3-be35-e8f931739aef service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:25.169901 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b272bdd-161a-47e3-be35-e8f931739aef service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:25.198805 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 979/1977] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:25 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:26:25.210084 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777af9c4-e61f-46a0-9dce-7848b51a0fe0 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:25.211580 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-777af9c4-e61f-46a0-9dce-7848b51a0fe0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:25.234814 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b272bdd-161a-47e3-be35-e8f931739aef service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:25.236255 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b272bdd-161a-47e3-be35-e8f931739aef service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:25.246866 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 999/1978] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:26:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 217 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:26:25.250290 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:25.279207 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777af9c4-e61f-46a0-9dce-7848b51a0fe0 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:25.282925 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee81d9c5-21ee-4169-9f01-11c70de738af service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:25.283561 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee81d9c5-21ee-4169-9f01-11c70de738af service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:25.284355 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee81d9c5-21ee-4169-9f01-11c70de738af service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:25.284627 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee81d9c5-21ee-4169-9f01-11c70de738af service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:25.293796 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee81d9c5-21ee-4169-9f01-11c70de738af service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:25.295380 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee81d9c5-21ee-4169-9f01-11c70de738af service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:25.306626 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 980/1979] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:26:25 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 27 01:26:25.388550 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b272bdd-161a-47e3-be35-e8f931739aef service neutron] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:25.400452 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b272bdd-161a-47e3-be35-e8f931739aef service neutron] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:25.405677 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560a2cb2-7e35-4751-bedc-c06988d28abf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:25.406297 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560a2cb2-7e35-4751-bedc-c06988d28abf service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:25.406638 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560a2cb2-7e35-4751-bedc-c06988d28abf service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:25.406917 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560a2cb2-7e35-4751-bedc-c06988d28abf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:25.409524 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-560a2cb2-7e35-4751-bedc-c06988d28abf service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:25.484190 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560a2cb2-7e35-4751-bedc-c06988d28abf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:25.485748 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560a2cb2-7e35-4751-bedc-c06988d28abf service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:25.490238 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:25.501823 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1000/1980] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:26:25 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:26:25.538953 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee81d9c5-21ee-4169-9f01-11c70de738af service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:25.540456 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ee81d9c5-21ee-4169-9f01-11c70de738af service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:25.558066 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee81d9c5-21ee-4169-9f01-11c70de738af service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:25.563645 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be90917b-22ed-4d6d-a46a-272f80eb82e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:25.563955 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be90917b-22ed-4d6d-a46a-272f80eb82e9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:25.564230 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be90917b-22ed-4d6d-a46a-272f80eb82e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:25.564507 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be90917b-22ed-4d6d-a46a-272f80eb82e9 admin admin] PATH_INFO: `/v3/projects/2a64b8cf904d450f9b85cbb97590d98e` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:25.565817 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be90917b-22ed-4d6d-a46a-272f80eb82e9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2a64b8cf904d450f9b85cbb97590d98e)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:25.571250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be90917b-22ed-4d6d-a46a-272f80eb82e9 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:25.647877 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-be90917b-22ed-4d6d-a46a-272f80eb82e9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2a64b8cf904d450f9b85cbb97590d98e', 'request_id': 'req-be90917b-22ed-4d6d-a46a-272f80eb82e9'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:25.660246 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-be90917b-22ed-4d6d-a46a-272f80eb82e9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2a64b8cf904d450f9b85cbb97590d98e', 'request_id': 'req-be90917b-22ed-4d6d-a46a-272f80eb82e9'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:25.674570 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-be90917b-22ed-4d6d-a46a-272f80eb82e9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2a64b8cf904d450f9b85cbb97590d98e', 'request_id': 'req-be90917b-22ed-4d6d-a46a-272f80eb82e9'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:25.688338 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-be90917b-22ed-4d6d-a46a-272f80eb82e9 admin admin] The token cache is being invalidate because project 2a64b8cf904d450f9b85cbb97590d98e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:26:25.688731 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-be90917b-22ed-4d6d-a46a-272f80eb82e9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:25.689478 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-be90917b-22ed-4d6d-a46a-272f80eb82e9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:25.691698 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 981/1981] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:26:25 2022] DELETE /identity/v3/projects/2a64b8cf904d450f9b85cbb97590d98e => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:25.695739 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560a2cb2-7e35-4751-bedc-c06988d28abf service cinder] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:25.707200 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560a2cb2-7e35-4751-bedc-c06988d28abf service cinder] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:25.780013 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adeb52f5-8cac-425c-8a84-01e3809a642c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:25.780906 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adeb52f5-8cac-425c-8a84-01e3809a642c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:25.781287 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adeb52f5-8cac-425c-8a84-01e3809a642c admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:25.781644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adeb52f5-8cac-425c-8a84-01e3809a642c admin admin] PATH_INFO: `/v3/users/596b58e918664148b8d97e9b94902bab` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:25.783343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adeb52f5-8cac-425c-8a84-01e3809a642c admin admin] RBAC: Authorizing `identity:delete_user(user_id=596b58e918664148b8d97e9b94902bab)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:25.788947 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adeb52f5-8cac-425c-8a84-01e3809a642c admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:25.859244 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-adeb52f5-8cac-425c-8a84-01e3809a642c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '596b58e918664148b8d97e9b94902bab'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:25.866947 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-adeb52f5-8cac-425c-8a84-01e3809a642c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '596b58e918664148b8d97e9b94902bab'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:25.887161 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-adeb52f5-8cac-425c-8a84-01e3809a642c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '596b58e918664148b8d97e9b94902bab'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:25.887980 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-adeb52f5-8cac-425c-8a84-01e3809a642c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '596b58e918664148b8d97e9b94902bab'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:25.888601 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-adeb52f5-8cac-425c-8a84-01e3809a642c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '596b58e918664148b8d97e9b94902bab'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:25.901467 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1001/1982] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:26:25 2022] DELETE /identity/v3/users/596b58e918664148b8d97e9b94902bab => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:26.215022 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be90917b-22ed-4d6d-a46a-272f80eb82e9 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:26.216736 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-be90917b-22ed-4d6d-a46a-272f80eb82e9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:26.231790 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be90917b-22ed-4d6d-a46a-272f80eb82e9 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:26.293643 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33cd8916-deb9-446a-a82f-0a5b5a1aefd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:26.294209 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cd8916-deb9-446a-a82f-0a5b5a1aefd5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:26.294490 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cd8916-deb9-446a-a82f-0a5b5a1aefd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:26.294761 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cd8916-deb9-446a-a82f-0a5b5a1aefd5 admin admin] PATH_INFO: `/v3/projects/f1a27696da84492e892d10e4bac4df38` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:26.295886 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33cd8916-deb9-446a-a82f-0a5b5a1aefd5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f1a27696da84492e892d10e4bac4df38)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:26.297901 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33cd8916-deb9-446a-a82f-0a5b5a1aefd5 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:26.345799 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-33cd8916-deb9-446a-a82f-0a5b5a1aefd5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f1a27696da84492e892d10e4bac4df38', 'request_id': 'req-33cd8916-deb9-446a-a82f-0a5b5a1aefd5'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:26.362328 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-33cd8916-deb9-446a-a82f-0a5b5a1aefd5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f1a27696da84492e892d10e4bac4df38', 'request_id': 'req-33cd8916-deb9-446a-a82f-0a5b5a1aefd5'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:26.364045 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:26.372015 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-33cd8916-deb9-446a-a82f-0a5b5a1aefd5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f1a27696da84492e892d10e4bac4df38', 'request_id': 'req-33cd8916-deb9-446a-a82f-0a5b5a1aefd5'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:26.387622 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-33cd8916-deb9-446a-a82f-0a5b5a1aefd5 admin admin] The token cache is being invalidate because project f1a27696da84492e892d10e4bac4df38 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:26:26.387931 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-33cd8916-deb9-446a-a82f-0a5b5a1aefd5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:26.388585 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-33cd8916-deb9-446a-a82f-0a5b5a1aefd5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:26.390506 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 982/1983] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:26:26 2022] DELETE /identity/v3/projects/f1a27696da84492e892d10e4bac4df38 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:26.394771 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adeb52f5-8cac-425c-8a84-01e3809a642c admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:26.396202 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-adeb52f5-8cac-425c-8a84-01e3809a642c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:26.409052 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adeb52f5-8cac-425c-8a84-01e3809a642c admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:26.470340 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc23732-bd7f-4347-b50b-97ddea6156be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:26.470978 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc23732-bd7f-4347-b50b-97ddea6156be admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:26.471305 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc23732-bd7f-4347-b50b-97ddea6156be admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:26.471722 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc23732-bd7f-4347-b50b-97ddea6156be admin admin] PATH_INFO: `/v3/users/8871f883f18f409cbb34817e7b80896c` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:26.473276 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc23732-bd7f-4347-b50b-97ddea6156be admin admin] RBAC: Authorizing `identity:delete_user(user_id=8871f883f18f409cbb34817e7b80896c)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:26.475282 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc23732-bd7f-4347-b50b-97ddea6156be admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:26.497342 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:26.524243 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-acc23732-bd7f-4347-b50b-97ddea6156be admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8871f883f18f409cbb34817e7b80896c'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:26.531543 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-acc23732-bd7f-4347-b50b-97ddea6156be admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8871f883f18f409cbb34817e7b80896c'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:26.543607 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-acc23732-bd7f-4347-b50b-97ddea6156be admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8871f883f18f409cbb34817e7b80896c'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:26.544306 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-acc23732-bd7f-4347-b50b-97ddea6156be admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8871f883f18f409cbb34817e7b80896c'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:26.544833 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-acc23732-bd7f-4347-b50b-97ddea6156be admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8871f883f18f409cbb34817e7b80896c'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:26.557593 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1002/1984] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:26:26 2022] DELETE /identity/v3/users/8871f883f18f409cbb34817e7b80896c => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:26.886044 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33cd8916-deb9-446a-a82f-0a5b5a1aefd5 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:26.887710 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-33cd8916-deb9-446a-a82f-0a5b5a1aefd5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:26.903790 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33cd8916-deb9-446a-a82f-0a5b5a1aefd5 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:26.958597 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc23732-bd7f-4347-b50b-97ddea6156be admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:26.960767 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-acc23732-bd7f-4347-b50b-97ddea6156be admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:27.003643 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753c1ca3-c336-48bb-b858-2c4f3c25d6c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:27.004292 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753c1ca3-c336-48bb-b858-2c4f3c25d6c6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:27.004576 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753c1ca3-c336-48bb-b858-2c4f3c25d6c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:27.004852 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753c1ca3-c336-48bb-b858-2c4f3c25d6c6 admin admin] PATH_INFO: `/v3/projects/698e3b45fa3543c58665717e84d251ea` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:27.006071 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753c1ca3-c336-48bb-b858-2c4f3c25d6c6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=698e3b45fa3543c58665717e84d251ea)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:27.008002 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753c1ca3-c336-48bb-b858-2c4f3c25d6c6 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:27.060015 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc23732-bd7f-4347-b50b-97ddea6156be admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:27.066836 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-753c1ca3-c336-48bb-b858-2c4f3c25d6c6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '698e3b45fa3543c58665717e84d251ea', 'request_id': 'req-753c1ca3-c336-48bb-b858-2c4f3c25d6c6'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:27.076896 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-753c1ca3-c336-48bb-b858-2c4f3c25d6c6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '698e3b45fa3543c58665717e84d251ea', 'request_id': 'req-753c1ca3-c336-48bb-b858-2c4f3c25d6c6'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:27.087165 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-753c1ca3-c336-48bb-b858-2c4f3c25d6c6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '698e3b45fa3543c58665717e84d251ea', 'request_id': 'req-753c1ca3-c336-48bb-b858-2c4f3c25d6c6'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:27.110429 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-753c1ca3-c336-48bb-b858-2c4f3c25d6c6 admin admin] The token cache is being invalidate because project 698e3b45fa3543c58665717e84d251ea was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:26:27.110575 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-753c1ca3-c336-48bb-b858-2c4f3c25d6c6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:27.116850 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-753c1ca3-c336-48bb-b858-2c4f3c25d6c6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:27.121298 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 983/1985] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:26:26 2022] DELETE /identity/v3/projects/698e3b45fa3543c58665717e84d251ea => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:27.125966 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a55763f-8239-485c-a4bb-1ba2c5d749cf None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:27.126904 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a55763f-8239-485c-a4bb-1ba2c5d749cf None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:27.129324 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a55763f-8239-485c-a4bb-1ba2c5d749cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:27.138877 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e72b8f94-593e-440d-84fd-db080c4c308c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:27.139645 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72b8f94-593e-440d-84fd-db080c4c308c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:27.140125 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72b8f94-593e-440d-84fd-db080c4c308c admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:27.140459 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72b8f94-593e-440d-84fd-db080c4c308c admin admin] PATH_INFO: `/v3/users/2c97470658cb463c838d494d1e4ad2a8` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:27.141714 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e72b8f94-593e-440d-84fd-db080c4c308c admin admin] RBAC: Authorizing `identity:delete_user(user_id=2c97470658cb463c838d494d1e4ad2a8)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:27.153021 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e72b8f94-593e-440d-84fd-db080c4c308c admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:27.165938 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-8a55763f-8239-485c-a4bb-1ba2c5d749cf None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:27.167146 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-8a55763f-8239-485c-a4bb-1ba2c5d749cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:27.175742 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 984/1986] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:26:27.183919 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9ed741-4406-41fc-b78f-82c429c67630 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:27.184236 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9ed741-4406-41fc-b78f-82c429c67630 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:27.184514 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9ed741-4406-41fc-b78f-82c429c67630 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:27.214131 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-8f9ed741-4406-41fc-b78f-82c429c67630 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:27.215838 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-8f9ed741-4406-41fc-b78f-82c429c67630 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:27.216409 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:DELETE=15,keystone:SELECT=84,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:27.219654 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-e72b8f94-593e-440d-84fd-db080c4c308c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2c97470658cb463c838d494d1e4ad2a8'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:27.230795 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-e72b8f94-593e-440d-84fd-db080c4c308c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2c97470658cb463c838d494d1e4ad2a8'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:27.242725 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 985/1987] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:26:27.248544 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9ed741-4406-41fc-b78f-82c429c67630 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:27.251775 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-e72b8f94-593e-440d-84fd-db080c4c308c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2c97470658cb463c838d494d1e4ad2a8'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:27.252476 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-e72b8f94-593e-440d-84fd-db080c4c308c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2c97470658cb463c838d494d1e4ad2a8'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:27.253160 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-e72b8f94-593e-440d-84fd-db080c4c308c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2c97470658cb463c838d494d1e4ad2a8'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:27.259023 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9ed741-4406-41fc-b78f-82c429c67630 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:27.263898 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d1f496-1a4c-4fa0-8ae0-a5cd91eb9806 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:27.264486 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d1f496-1a4c-4fa0-8ae0-a5cd91eb9806 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:27.264762 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d1f496-1a4c-4fa0-8ae0-a5cd91eb9806 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:27.265032 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d1f496-1a4c-4fa0-8ae0-a5cd91eb9806 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:27.265639 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1003/1988] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:26:26 2022] DELETE /identity/v3/users/2c97470658cb463c838d494d1e4ad2a8 => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:27.265747 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06d1f496-1a4c-4fa0-8ae0-a5cd91eb9806 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:26:27.266067 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06d1f496-1a4c-4fa0-8ae0-a5cd91eb9806 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:27.267689 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06d1f496-1a4c-4fa0-8ae0-a5cd91eb9806 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:27.270540 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2ca94a-29ac-432d-9dc9-8faf3ab86671 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:27.270824 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2ca94a-29ac-432d-9dc9-8faf3ab86671 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:27.271096 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2ca94a-29ac-432d-9dc9-8faf3ab86671 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:27.280044 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 986/1989] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:26:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:26:27.284971 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d1f496-1a4c-4fa0-8ae0-a5cd91eb9806 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:27.286802 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-06d1f496-1a4c-4fa0-8ae0-a5cd91eb9806 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:27.291133 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-ef2ca94a-29ac-432d-9dc9-8faf3ab86671 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:27.292288 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-ef2ca94a-29ac-432d-9dc9-8faf3ab86671 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:27.301582 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d1f496-1a4c-4fa0-8ae0-a5cd91eb9806 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:27.348040 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:DELETE=9,keystone:SELECT=90,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:27.363839 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1004/1990] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:26:27.366493 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dca77e0-4bd3-41c3-833f-03b9567d26c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:27.367137 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dca77e0-4bd3-41c3-833f-03b9567d26c8 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:27.367397 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dca77e0-4bd3-41c3-833f-03b9567d26c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:27.367659 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dca77e0-4bd3-41c3-833f-03b9567d26c8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:27.368295 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dca77e0-4bd3-41c3-833f-03b9567d26c8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:27.370385 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef2ca94a-29ac-432d-9dc9-8faf3ab86671 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:27.370517 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dca77e0-4bd3-41c3-833f-03b9567d26c8 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:27.371738 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-ef2ca94a-29ac-432d-9dc9-8faf3ab86671 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:27.420978 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 987/1991] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:26:27 2022] POST /identity/v3/projects => generated 365 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:26:27.426951 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dca77e0-4bd3-41c3-833f-03b9567d26c8 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:27.449660 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef2ca94a-29ac-432d-9dc9-8faf3ab86671 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:27.506797 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6263f71f-edf8-4e90-9976-6781eccb15fc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:27.507434 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6263f71f-edf8-4e90-9976-6781eccb15fc service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:27.507693 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6263f71f-edf8-4e90-9976-6781eccb15fc service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:27.507965 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6263f71f-edf8-4e90-9976-6781eccb15fc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:27.520994 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dca77e0-4bd3-41c3-833f-03b9567d26c8 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:27.526126 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044bdcd6-f3c1-4ecc-b3dd-fa23a8f09648 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:27.526748 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044bdcd6-f3c1-4ecc-b3dd-fa23a8f09648 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:27.527084 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044bdcd6-f3c1-4ecc-b3dd-fa23a8f09648 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:27.527390 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044bdcd6-f3c1-4ecc-b3dd-fa23a8f09648 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:27.528035 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044bdcd6-f3c1-4ecc-b3dd-fa23a8f09648 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:27.530211 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044bdcd6-f3c1-4ecc-b3dd-fa23a8f09648 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:27.544126 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:27.553200 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 988/1992] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:26:27 2022] POST /identity/v3/users => generated 283 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:26:27.557646 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044bdcd6-f3c1-4ecc-b3dd-fa23a8f09648 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:27.568252 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044bdcd6-f3c1-4ecc-b3dd-fa23a8f09648 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:27.573116 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc501418-b6f3-48fd-9c47-4c2d2c42b23b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:27.574229 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc501418-b6f3-48fd-9c47-4c2d2c42b23b admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:27.574513 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc501418-b6f3-48fd-9c47-4c2d2c42b23b admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:27.574785 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc501418-b6f3-48fd-9c47-4c2d2c42b23b admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:27.575577 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc501418-b6f3-48fd-9c47-4c2d2c42b23b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:27.575830 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6263f71f-edf8-4e90-9976-6781eccb15fc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:27.577194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc501418-b6f3-48fd-9c47-4c2d2c42b23b admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:27.577339 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6263f71f-edf8-4e90-9976-6781eccb15fc service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:27.587633 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 989/1993] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:26:27 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:26:27.591875 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc501418-b6f3-48fd-9c47-4c2d2c42b23b admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:27.593105 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1005/1994] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:26:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 221 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 27 01:26:27.604484 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc501418-b6f3-48fd-9c47-4c2d2c42b23b admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:27.609147 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14b3d6c-4e2a-4d7a-823f-0b898cff0f06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:27.609805 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14b3d6c-4e2a-4d7a-823f-0b898cff0f06 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:27.610109 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14b3d6c-4e2a-4d7a-823f-0b898cff0f06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:27.610388 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14b3d6c-4e2a-4d7a-823f-0b898cff0f06 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:27.611025 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14b3d6c-4e2a-4d7a-823f-0b898cff0f06 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:27.612675 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14b3d6c-4e2a-4d7a-823f-0b898cff0f06 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:27.623958 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 990/1995] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:26:27 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 27 01:26:27.628775 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6263f71f-edf8-4e90-9976-6781eccb15fc service neutron] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:27.639438 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6263f71f-edf8-4e90-9976-6781eccb15fc service neutron] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:27.645568 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14bf187b-5ab8-4091-b777-1106bba0e666 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:27.646305 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14bf187b-5ab8-4091-b777-1106bba0e666 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:27.646622 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14bf187b-5ab8-4091-b777-1106bba0e666 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:27.646921 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14bf187b-5ab8-4091-b777-1106bba0e666 admin admin] PATH_INFO: `/v3/projects/6173634a570446f48fe4bf89887626df/users/0ca2f266188f4001abf68776b42f2f82/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:27.666191 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14bf187b-5ab8-4091-b777-1106bba0e666 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6173634a570446f48fe4bf89887626df, user_id=0ca2f266188f4001abf68776b42f2f82, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:27.669036 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14bf187b-5ab8-4091-b777-1106bba0e666 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:27.684729 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1006/1996] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:26:27 2022] PUT /identity/v3/projects/6173634a570446f48fe4bf89887626df/users/0ca2f266188f4001abf68776b42f2f82/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:27.758819 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97db885-2751-437a-98c4-ee54a7c93e26 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:27.759108 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97db885-2751-437a-98c4-ee54a7c93e26 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:27.759318 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97db885-2751-437a-98c4-ee54a7c93e26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:27.787990 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-f97db885-2751-437a-98c4-ee54a7c93e26 None None] MFA Rules not processed for user `0ca2f266188f4001abf68776b42f2f82`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:27.789260 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f97db885-2751-437a-98c4-ee54a7c93e26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:27.869747 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 991/1997] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:27 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:27.881211 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14bf187b-5ab8-4091-b777-1106bba0e666 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:27.883146 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-14bf187b-5ab8-4091-b777-1106bba0e666 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:27.890304 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97db885-2751-437a-98c4-ee54a7c93e26 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:27.891885 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f97db885-2751-437a-98c4-ee54a7c93e26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:27.952852 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:27.975198 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14bf187b-5ab8-4091-b777-1106bba0e666 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:27.978953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e21a0b4-2684-4d62-9eec-7a8ddbb8045a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:27.979794 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e21a0b4-2684-4d62-9eec-7a8ddbb8045a service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:27.979939 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e21a0b4-2684-4d62-9eec-7a8ddbb8045a service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:27.980311 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e21a0b4-2684-4d62-9eec-7a8ddbb8045a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:27.989335 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e21a0b4-2684-4d62-9eec-7a8ddbb8045a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:27.990774 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e21a0b4-2684-4d62-9eec-7a8ddbb8045a service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:27.996753 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97db885-2751-437a-98c4-ee54a7c93e26 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:28.002858 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1007/1998] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:26:27 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:26:28.005297 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5840dc2-fa3e-4913-b0d2-62798ee43437 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:28.005950 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5840dc2-fa3e-4913-b0d2-62798ee43437 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:28.006227 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5840dc2-fa3e-4913-b0d2-62798ee43437 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:28.006501 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5840dc2-fa3e-4913-b0d2-62798ee43437 admin admin] PATH_INFO: `/v3/projects/a5b460240ea649b0b12d8b9387003dc8` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:28.007631 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5840dc2-fa3e-4913-b0d2-62798ee43437 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a5b460240ea649b0b12d8b9387003dc8)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:28.009683 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5840dc2-fa3e-4913-b0d2-62798ee43437 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:28.070040 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d5840dc2-fa3e-4913-b0d2-62798ee43437 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a5b460240ea649b0b12d8b9387003dc8', 'request_id': 'req-d5840dc2-fa3e-4913-b0d2-62798ee43437'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:28.080309 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d5840dc2-fa3e-4913-b0d2-62798ee43437 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a5b460240ea649b0b12d8b9387003dc8', 'request_id': 'req-d5840dc2-fa3e-4913-b0d2-62798ee43437'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:28.087269 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d5840dc2-fa3e-4913-b0d2-62798ee43437 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a5b460240ea649b0b12d8b9387003dc8', 'request_id': 'req-d5840dc2-fa3e-4913-b0d2-62798ee43437'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:28.103462 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d5840dc2-fa3e-4913-b0d2-62798ee43437 admin admin] The token cache is being invalidate because project a5b460240ea649b0b12d8b9387003dc8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:26:28.103462 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d5840dc2-fa3e-4913-b0d2-62798ee43437 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:28.103978 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d5840dc2-fa3e-4913-b0d2-62798ee43437 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:28.106973 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 992/1999] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:26:27 2022] DELETE /identity/v3/projects/a5b460240ea649b0b12d8b9387003dc8 => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:26:33.047415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7101ca15-6478-4314-b5c1-b32262628e73 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:33.047918 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7101ca15-6478-4314-b5c1-b32262628e73 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:33.048027 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7101ca15-6478-4314-b5c1-b32262628e73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:33.068632 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-7101ca15-6478-4314-b5c1-b32262628e73 None None] MFA Rules not processed for user `1505f37c15c0461a912d59172ebdf94a`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:33.069886 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-7101ca15-6478-4314-b5c1-b32262628e73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:33.134643 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1008/2000] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:33 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:33.917998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48bb3328-569b-4442-9cd8-b9dd8a8a36fa None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:33.918547 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48bb3328-569b-4442-9cd8-b9dd8a8a36fa None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:33.918693 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48bb3328-569b-4442-9cd8-b9dd8a8a36fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:33.941617 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-48bb3328-569b-4442-9cd8-b9dd8a8a36fa None None] MFA Rules not processed for user `d311fc1f82124819b509f766fa432e2c`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:33.944092 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-48bb3328-569b-4442-9cd8-b9dd8a8a36fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:34.034757 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 993/2001] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:33 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:43.127505 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=75,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:44.026199 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=62,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:47.651520 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7101ca15-6478-4314-b5c1-b32262628e73 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:47.652456 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-7101ca15-6478-4314-b5c1-b32262628e73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:47.730893 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7101ca15-6478-4314-b5c1-b32262628e73 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:47.736101 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6afee038-e57f-4c72-9f2e-923a0e28823d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:47.736685 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afee038-e57f-4c72-9f2e-923a0e28823d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:47.736960 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afee038-e57f-4c72-9f2e-923a0e28823d admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:47.737305 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afee038-e57f-4c72-9f2e-923a0e28823d admin admin] PATH_INFO: `/v3/users/f4da0cda7fbd4ede8d3051a4544c2e8c` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:47.738460 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6afee038-e57f-4c72-9f2e-923a0e28823d admin admin] RBAC: Authorizing `identity:delete_user(user_id=f4da0cda7fbd4ede8d3051a4544c2e8c)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:47.740422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6afee038-e57f-4c72-9f2e-923a0e28823d admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:47.806155 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-6afee038-e57f-4c72-9f2e-923a0e28823d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f4da0cda7fbd4ede8d3051a4544c2e8c'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:47.819534 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-6afee038-e57f-4c72-9f2e-923a0e28823d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f4da0cda7fbd4ede8d3051a4544c2e8c'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:47.833298 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-6afee038-e57f-4c72-9f2e-923a0e28823d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f4da0cda7fbd4ede8d3051a4544c2e8c'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:47.834008 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-6afee038-e57f-4c72-9f2e-923a0e28823d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f4da0cda7fbd4ede8d3051a4544c2e8c'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:47.834721 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-6afee038-e57f-4c72-9f2e-923a0e28823d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f4da0cda7fbd4ede8d3051a4544c2e8c'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:47.848688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1009/2002] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:26:47 2022] DELETE /identity/v3/users/f4da0cda7fbd4ede8d3051a4544c2e8c => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:47.853295 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06db0f85-32b6-43ef-baed-6e11eaeb5822 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:47.853777 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06db0f85-32b6-43ef-baed-6e11eaeb5822 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:47.853908 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06db0f85-32b6-43ef-baed-6e11eaeb5822 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:47.874603 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-06db0f85-32b6-43ef-baed-6e11eaeb5822 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:47.875801 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-06db0f85-32b6-43ef-baed-6e11eaeb5822 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:47.947988 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 994/2003] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:47.958521 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6afee038-e57f-4c72-9f2e-923a0e28823d admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:47.960188 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-6afee038-e57f-4c72-9f2e-923a0e28823d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:48.024922 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6afee038-e57f-4c72-9f2e-923a0e28823d admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:48.028683 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e0e4c1-4b63-43f4-ae7b-5916ec76982b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:48.029330 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e0e4c1-4b63-43f4-ae7b-5916ec76982b service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:48.029602 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e0e4c1-4b63-43f4-ae7b-5916ec76982b service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:48.029874 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e0e4c1-4b63-43f4-ae7b-5916ec76982b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:48.039200 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e0e4c1-4b63-43f4-ae7b-5916ec76982b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:48.040624 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e0e4c1-4b63-43f4-ae7b-5916ec76982b service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:48.053128 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1010/2004] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:26:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:26:48.389949 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06db0f85-32b6-43ef-baed-6e11eaeb5822 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:48.391501 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-06db0f85-32b6-43ef-baed-6e11eaeb5822 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:48.405596 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06db0f85-32b6-43ef-baed-6e11eaeb5822 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:48.420453 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e782255-31f3-4b1c-8212-37c0bb5e987a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:48.421252 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e782255-31f3-4b1c-8212-37c0bb5e987a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:48.422053 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e782255-31f3-4b1c-8212-37c0bb5e987a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:48.422481 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e782255-31f3-4b1c-8212-37c0bb5e987a admin admin] PATH_INFO: `/v3/projects/9879b9937ead4ef3a6c1c9de77d3a63d` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:48.423749 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e782255-31f3-4b1c-8212-37c0bb5e987a admin admin] RBAC: Authorizing `identity:delete_project(project_id=9879b9937ead4ef3a6c1c9de77d3a63d)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:48.426295 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e782255-31f3-4b1c-8212-37c0bb5e987a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:48.523582 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4e782255-31f3-4b1c-8212-37c0bb5e987a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9879b9937ead4ef3a6c1c9de77d3a63d', 'request_id': 'req-4e782255-31f3-4b1c-8212-37c0bb5e987a'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:48.532132 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4e782255-31f3-4b1c-8212-37c0bb5e987a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9879b9937ead4ef3a6c1c9de77d3a63d', 'request_id': 'req-4e782255-31f3-4b1c-8212-37c0bb5e987a'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:48.538268 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4e782255-31f3-4b1c-8212-37c0bb5e987a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9879b9937ead4ef3a6c1c9de77d3a63d', 'request_id': 'req-4e782255-31f3-4b1c-8212-37c0bb5e987a'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:48.549896 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4e782255-31f3-4b1c-8212-37c0bb5e987a admin admin] The token cache is being invalidate because project 9879b9937ead4ef3a6c1c9de77d3a63d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:26:48.550182 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4e782255-31f3-4b1c-8212-37c0bb5e987a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:48.550844 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4e782255-31f3-4b1c-8212-37c0bb5e987a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:48.552543 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 995/2005] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:26:48 2022] DELETE /identity/v3/projects/9879b9937ead4ef3a6c1c9de77d3a63d => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:48.556862 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e0e4c1-4b63-43f4-ae7b-5916ec76982b service neutron] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:48.567375 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e0e4c1-4b63-43f4-ae7b-5916ec76982b service neutron] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:48.587557 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:48.628618 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3577251f-4035-4ee6-8408-a7185f31f420 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:48.629269 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3577251f-4035-4ee6-8408-a7185f31f420 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:48.629546 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3577251f-4035-4ee6-8408-a7185f31f420 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:48.629814 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3577251f-4035-4ee6-8408-a7185f31f420 admin admin] PATH_INFO: `/v3/users/917d19d1f44c42969e0efccd0f11019a` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:48.630903 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3577251f-4035-4ee6-8408-a7185f31f420 admin admin] RBAC: Authorizing `identity:delete_user(user_id=917d19d1f44c42969e0efccd0f11019a)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:48.632803 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3577251f-4035-4ee6-8408-a7185f31f420 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:48.682607 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-3577251f-4035-4ee6-8408-a7185f31f420 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '917d19d1f44c42969e0efccd0f11019a'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:48.690931 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-3577251f-4035-4ee6-8408-a7185f31f420 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '917d19d1f44c42969e0efccd0f11019a'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:48.703738 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-3577251f-4035-4ee6-8408-a7185f31f420 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '917d19d1f44c42969e0efccd0f11019a'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:48.704399 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-3577251f-4035-4ee6-8408-a7185f31f420 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '917d19d1f44c42969e0efccd0f11019a'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:48.705888 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-3577251f-4035-4ee6-8408-a7185f31f420 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '917d19d1f44c42969e0efccd0f11019a'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:48.718749 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1011/2006] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:26:48 2022] DELETE /identity/v3/users/917d19d1f44c42969e0efccd0f11019a => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:26:49.072488 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e782255-31f3-4b1c-8212-37c0bb5e987a admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:49.073980 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-4e782255-31f3-4b1c-8212-37c0bb5e987a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:49.088004 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e782255-31f3-4b1c-8212-37c0bb5e987a admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:49.158551 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a48bf0c-4914-443a-83ea-f719476e1ab6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:49.159269 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a48bf0c-4914-443a-83ea-f719476e1ab6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:49.159548 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a48bf0c-4914-443a-83ea-f719476e1ab6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:49.159819 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a48bf0c-4914-443a-83ea-f719476e1ab6 admin admin] PATH_INFO: `/v3/projects/d1b95477ea8e456bafe9470310b4ce3b` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:49.160949 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a48bf0c-4914-443a-83ea-f719476e1ab6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d1b95477ea8e456bafe9470310b4ce3b)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:49.162907 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a48bf0c-4914-443a-83ea-f719476e1ab6 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:49.194474 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:49.213015 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3a48bf0c-4914-443a-83ea-f719476e1ab6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd1b95477ea8e456bafe9470310b4ce3b', 'request_id': 'req-3a48bf0c-4914-443a-83ea-f719476e1ab6'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:49.221804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3a48bf0c-4914-443a-83ea-f719476e1ab6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd1b95477ea8e456bafe9470310b4ce3b', 'request_id': 'req-3a48bf0c-4914-443a-83ea-f719476e1ab6'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:49.227706 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3a48bf0c-4914-443a-83ea-f719476e1ab6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd1b95477ea8e456bafe9470310b4ce3b', 'request_id': 'req-3a48bf0c-4914-443a-83ea-f719476e1ab6'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:49.238840 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3a48bf0c-4914-443a-83ea-f719476e1ab6 admin admin] The token cache is being invalidate because project d1b95477ea8e456bafe9470310b4ce3b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:26:49.239127 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3a48bf0c-4914-443a-83ea-f719476e1ab6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:49.239779 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3a48bf0c-4914-443a-83ea-f719476e1ab6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:49.241339 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 996/2007] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:26:49 2022] DELETE /identity/v3/projects/d1b95477ea8e456bafe9470310b4ce3b => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:49.246159 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96abd5e9-703e-4aee-8a9e-0a9db9849f79 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:49.246441 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96abd5e9-703e-4aee-8a9e-0a9db9849f79 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:49.246710 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96abd5e9-703e-4aee-8a9e-0a9db9849f79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:49.268193 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-96abd5e9-703e-4aee-8a9e-0a9db9849f79 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:49.269425 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-96abd5e9-703e-4aee-8a9e-0a9db9849f79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:49.342327 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1012/2008] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:49.350740 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9a33eb-e36f-433e-92e4-988139293665 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:49.351023 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9a33eb-e36f-433e-92e4-988139293665 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:49.351293 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9a33eb-e36f-433e-92e4-988139293665 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:49.377263 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-ac9a33eb-e36f-433e-92e4-988139293665 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:49.378517 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ac9a33eb-e36f-433e-92e4-988139293665 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:49.386660 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 997/2009] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:49.391039 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96abd5e9-703e-4aee-8a9e-0a9db9849f79 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:49.411227 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96abd5e9-703e-4aee-8a9e-0a9db9849f79 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:49.417487 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c008213-f439-4149-b364-38a667e68b78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:49.417689 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c008213-f439-4149-b364-38a667e68b78 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:49.418024 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c008213-f439-4149-b364-38a667e68b78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:49.418549 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c008213-f439-4149-b364-38a667e68b78 admin admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:49.419714 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c008213-f439-4149-b364-38a667e68b78 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:26:49.420090 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c008213-f439-4149-b364-38a667e68b78 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:49.422085 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c008213-f439-4149-b364-38a667e68b78 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:49.441266 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1013/2010] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:26:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:26:49.442562 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9a33eb-e36f-433e-92e4-988139293665 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:49.452949 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9a33eb-e36f-433e-92e4-988139293665 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:49.461627 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c15238-897e-4b3b-b6e1-44b38a3d588a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:49.462990 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c15238-897e-4b3b-b6e1-44b38a3d588a admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:49.465614 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c15238-897e-4b3b-b6e1-44b38a3d588a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:49.465803 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c15238-897e-4b3b-b6e1-44b38a3d588a admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:49.466899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c15238-897e-4b3b-b6e1-44b38a3d588a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:49.469226 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c15238-897e-4b3b-b6e1-44b38a3d588a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:49.513106 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 998/2011] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:26:49 2022] POST /identity/v3/projects => generated 385 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:26:49.516688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c008213-f439-4149-b364-38a667e68b78 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:49.542688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c008213-f439-4149-b364-38a667e68b78 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:49.590558 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:49.626286 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9681958-71af-4c95-919d-491fa5d4f4e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:49.627548 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9681958-71af-4c95-919d-491fa5d4f4e0 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:49.627793 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9681958-71af-4c95-919d-491fa5d4f4e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:49.628058 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9681958-71af-4c95-919d-491fa5d4f4e0 admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:49.628682 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9681958-71af-4c95-919d-491fa5d4f4e0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:49.630622 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9681958-71af-4c95-919d-491fa5d4f4e0 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:49.652168 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1014/2012] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:26:49 2022] POST /identity/v3/users => generated 293 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:26:49.656321 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c15238-897e-4b3b-b6e1-44b38a3d588a admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:49.668932 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c15238-897e-4b3b-b6e1-44b38a3d588a admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:49.674496 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449f164e-7947-43c9-b02b-dd16f609e6f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:49.675093 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449f164e-7947-43c9-b02b-dd16f609e6f8 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:49.675372 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449f164e-7947-43c9-b02b-dd16f609e6f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:49.675673 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449f164e-7947-43c9-b02b-dd16f609e6f8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:49.676359 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449f164e-7947-43c9-b02b-dd16f609e6f8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:49.678219 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449f164e-7947-43c9-b02b-dd16f609e6f8 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:49.689037 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 999/2013] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:26:49 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 27 01:26:49.693541 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9681958-71af-4c95-919d-491fa5d4f4e0 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:49.703425 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9681958-71af-4c95-919d-491fa5d4f4e0 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:49.708119 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98512399-3a53-4fa0-8200-c88f62286799 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:49.708706 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98512399-3a53-4fa0-8200-c88f62286799 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:49.709009 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98512399-3a53-4fa0-8200-c88f62286799 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:49.709444 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98512399-3a53-4fa0-8200-c88f62286799 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:49.710358 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98512399-3a53-4fa0-8200-c88f62286799 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:49.712148 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98512399-3a53-4fa0-8200-c88f62286799 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:49.726723 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1015/2014] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:26:49 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:26:49.728393 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449f164e-7947-43c9-b02b-dd16f609e6f8 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:49.738845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449f164e-7947-43c9-b02b-dd16f609e6f8 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:49.743389 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5a0c6d-aa96-4e88-8a8c-680bac5570e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:49.744031 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5a0c6d-aa96-4e88-8a8c-680bac5570e1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:49.744312 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5a0c6d-aa96-4e88-8a8c-680bac5570e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:49.744586 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5a0c6d-aa96-4e88-8a8c-680bac5570e1 admin admin] PATH_INFO: `/v3/projects/04bd52fbfb294ed194fb49a25c5c1ddd/users/ed4dbcd148e4483e8db7e45ff3a409d4/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:49.760433 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5a0c6d-aa96-4e88-8a8c-680bac5570e1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=04bd52fbfb294ed194fb49a25c5c1ddd, user_id=ed4dbcd148e4483e8db7e45ff3a409d4, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:49.763230 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5a0c6d-aa96-4e88-8a8c-680bac5570e1 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:49.776433 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1000/2015] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:26:49 2022] PUT /identity/v3/projects/04bd52fbfb294ed194fb49a25c5c1ddd/users/ed4dbcd148e4483e8db7e45ff3a409d4/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:49.786727 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4ebd32-729b-4119-8dbb-23c00ba5d121 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:49.787015 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4ebd32-729b-4119-8dbb-23c00ba5d121 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:49.787317 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4ebd32-729b-4119-8dbb-23c00ba5d121 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:49.819915 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-dd4ebd32-729b-4119-8dbb-23c00ba5d121 None None] MFA Rules not processed for user `ed4dbcd148e4483e8db7e45ff3a409d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:49.821137 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-dd4ebd32-729b-4119-8dbb-23c00ba5d121 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:49.886394 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1016/2016] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:49 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:49.890921 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5a0c6d-aa96-4e88-8a8c-680bac5570e1 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:49.954067 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5a0c6d-aa96-4e88-8a8c-680bac5570e1 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:49.959059 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f46a263-76a3-4be0-9c14-9fc42681e7a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:49.959673 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f46a263-76a3-4be0-9c14-9fc42681e7a2 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:49.959949 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f46a263-76a3-4be0-9c14-9fc42681e7a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:49.960238 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f46a263-76a3-4be0-9c14-9fc42681e7a2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:49.960868 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f46a263-76a3-4be0-9c14-9fc42681e7a2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:49.962815 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f46a263-76a3-4be0-9c14-9fc42681e7a2 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:49.990801 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1001/2017] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:26:49 2022] POST /identity/v3/projects => generated 387 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:26:49.994895 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4ebd32-729b-4119-8dbb-23c00ba5d121 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:50.005634 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4ebd32-729b-4119-8dbb-23c00ba5d121 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:50.063092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96441941-4795-4dde-a754-daec583e06f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:50.063712 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96441941-4795-4dde-a754-daec583e06f5 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:50.064600 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96441941-4795-4dde-a754-daec583e06f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:50.065031 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96441941-4795-4dde-a754-daec583e06f5 admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:50.065715 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96441941-4795-4dde-a754-daec583e06f5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:50.067684 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96441941-4795-4dde-a754-daec583e06f5 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:50.092383 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1017/2018] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:26:49 2022] POST /identity/v3/users => generated 300 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:26:50.093609 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f46a263-76a3-4be0-9c14-9fc42681e7a2 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:50.105668 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f46a263-76a3-4be0-9c14-9fc42681e7a2 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:50.110324 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515e892c-4873-4314-af2d-4372cc7ceefa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:50.111535 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515e892c-4873-4314-af2d-4372cc7ceefa admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:50.112375 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515e892c-4873-4314-af2d-4372cc7ceefa admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:50.112647 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515e892c-4873-4314-af2d-4372cc7ceefa admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:50.113316 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515e892c-4873-4314-af2d-4372cc7ceefa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:50.114901 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515e892c-4873-4314-af2d-4372cc7ceefa admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:50.128857 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1002/2019] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:26:50 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:26:50.130283 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96441941-4795-4dde-a754-daec583e06f5 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:50.141105 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96441941-4795-4dde-a754-daec583e06f5 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:50.146257 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a1b74b-9629-4c95-a59f-5adafe53f6ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:50.146925 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a1b74b-9629-4c95-a59f-5adafe53f6ae admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:50.147199 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a1b74b-9629-4c95-a59f-5adafe53f6ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:50.147469 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a1b74b-9629-4c95-a59f-5adafe53f6ae admin admin] PATH_INFO: `/v3/projects/c1b062a7917a4a788214b2d85f132c1e/users/d9542cc358ab4b40b180105d5823cabf/roles/414286927fdd44688107ff5e10235acf` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:50.164280 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a1b74b-9629-4c95-a59f-5adafe53f6ae admin admin] RBAC: Authorizing `identity:create_grant(project_id=c1b062a7917a4a788214b2d85f132c1e, user_id=d9542cc358ab4b40b180105d5823cabf, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:50.166986 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a1b74b-9629-4c95-a59f-5adafe53f6ae admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:50.182976 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1018/2020] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:26:50 2022] PUT /identity/v3/projects/c1b062a7917a4a788214b2d85f132c1e/users/d9542cc358ab4b40b180105d5823cabf/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:50.186796 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515e892c-4873-4314-af2d-4372cc7ceefa admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:50.197707 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515e892c-4873-4314-af2d-4372cc7ceefa admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:50.221901 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:50.263770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e00a0d-1ee4-416b-92a5-fa8f633b822d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:50.264379 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e00a0d-1ee4-416b-92a5-fa8f633b822d admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:50.264654 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e00a0d-1ee4-416b-92a5-fa8f633b822d admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:50.264922 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e00a0d-1ee4-416b-92a5-fa8f633b822d admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:50.265630 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e00a0d-1ee4-416b-92a5-fa8f633b822d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:50.267843 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e00a0d-1ee4-416b-92a5-fa8f633b822d admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:50.278415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1003/2021] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:26:50 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:26:50.283434 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a1b74b-9629-4c95-a59f-5adafe53f6ae admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:50.288794 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:50.295587 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a1b74b-9629-4c95-a59f-5adafe53f6ae admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:50.300708 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95944355-9c74-41b8-bea0-6a07a5cb5ef2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:50.301412 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95944355-9c74-41b8-bea0-6a07a5cb5ef2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:50.301688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95944355-9c74-41b8-bea0-6a07a5cb5ef2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:50.301962 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95944355-9c74-41b8-bea0-6a07a5cb5ef2 admin admin] PATH_INFO: `/v3/projects/c1b062a7917a4a788214b2d85f132c1e/users/d9542cc358ab4b40b180105d5823cabf/roles/414286927fdd44688107ff5e10235acf` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:50.303989 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95944355-9c74-41b8-bea0-6a07a5cb5ef2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c1b062a7917a4a788214b2d85f132c1e, user_id=d9542cc358ab4b40b180105d5823cabf, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:50.306961 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95944355-9c74-41b8-bea0-6a07a5cb5ef2 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:50.318313 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1019/2022] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:26:50 2022] PUT /identity/v3/projects/c1b062a7917a4a788214b2d85f132c1e/users/d9542cc358ab4b40b180105d5823cabf/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:50.329382 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5ed05d-8941-4eae-b5b7-e72a4c7ad82a None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:50.330264 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5ed05d-8941-4eae-b5b7-e72a4c7ad82a None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:50.330701 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5ed05d-8941-4eae-b5b7-e72a4c7ad82a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:50.358963 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-0c5ed05d-8941-4eae-b5b7-e72a4c7ad82a None None] MFA Rules not processed for user `d9542cc358ab4b40b180105d5823cabf`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:50.360188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0c5ed05d-8941-4eae-b5b7-e72a4c7ad82a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:50.430371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1004/2023] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:50 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:50.442654 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95944355-9c74-41b8-bea0-6a07a5cb5ef2 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:50.444423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-95944355-9c74-41b8-bea0-6a07a5cb5ef2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:50.504484 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95944355-9c74-41b8-bea0-6a07a5cb5ef2 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:50.508173 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2661a6c6-de36-4da3-9e5b-73915073705d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:50.508784 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2661a6c6-de36-4da3-9e5b-73915073705d service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:50.509140 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2661a6c6-de36-4da3-9e5b-73915073705d service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:50.509358 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2661a6c6-de36-4da3-9e5b-73915073705d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:50.519226 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2661a6c6-de36-4da3-9e5b-73915073705d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:50.522020 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2661a6c6-de36-4da3-9e5b-73915073705d service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:50.534149 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1020/2024] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:26:50 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:26:50.565588 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5ed05d-8941-4eae-b5b7-e72a4c7ad82a None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:50.639573 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5ed05d-8941-4eae-b5b7-e72a4c7ad82a None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:50.644853 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2fdafba-a8ed-4b06-b12b-60b9c02e4bbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:50.645540 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2fdafba-a8ed-4b06-b12b-60b9c02e4bbd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:50.645807 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2fdafba-a8ed-4b06-b12b-60b9c02e4bbd admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:50.646076 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2fdafba-a8ed-4b06-b12b-60b9c02e4bbd admin admin] PATH_INFO: `/v3/users/ed4dbcd148e4483e8db7e45ff3a409d4` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:50.649888 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2fdafba-a8ed-4b06-b12b-60b9c02e4bbd admin admin] RBAC: Authorizing `identity:delete_user(user_id=ed4dbcd148e4483e8db7e45ff3a409d4)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:50.652145 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2fdafba-a8ed-4b06-b12b-60b9c02e4bbd admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:50.705013 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d2fdafba-a8ed-4b06-b12b-60b9c02e4bbd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ed4dbcd148e4483e8db7e45ff3a409d4'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:50.706721 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:50.713961 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d2fdafba-a8ed-4b06-b12b-60b9c02e4bbd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ed4dbcd148e4483e8db7e45ff3a409d4'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:50.714919 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d2fdafba-a8ed-4b06-b12b-60b9c02e4bbd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ed4dbcd148e4483e8db7e45ff3a409d4'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:50.727955 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d2fdafba-a8ed-4b06-b12b-60b9c02e4bbd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ed4dbcd148e4483e8db7e45ff3a409d4'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:50.734771 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d2fdafba-a8ed-4b06-b12b-60b9c02e4bbd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ed4dbcd148e4483e8db7e45ff3a409d4'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:50.741335 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1005/2025] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:26:50 2022] DELETE /identity/v3/users/ed4dbcd148e4483e8db7e45ff3a409d4 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:50.746322 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d831d6e9-22eb-4543-868e-5e1fcc587e6e None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:50.746802 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d831d6e9-22eb-4543-868e-5e1fcc587e6e None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:50.747008 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d831d6e9-22eb-4543-868e-5e1fcc587e6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:50.766954 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-d831d6e9-22eb-4543-868e-5e1fcc587e6e None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:50.768184 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-d831d6e9-22eb-4543-868e-5e1fcc587e6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:50.828303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1021/2026] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:26:50.838154 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2fdafba-a8ed-4b06-b12b-60b9c02e4bbd admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:50.839547 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d2fdafba-a8ed-4b06-b12b-60b9c02e4bbd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:50.899279 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2fdafba-a8ed-4b06-b12b-60b9c02e4bbd admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:50.903019 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1b43f61-5e1d-4f3c-af28-896ea269150a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:50.903683 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b43f61-5e1d-4f3c-af28-896ea269150a service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:50.903970 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b43f61-5e1d-4f3c-af28-896ea269150a service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:50.904249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b43f61-5e1d-4f3c-af28-896ea269150a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:50.913795 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1b43f61-5e1d-4f3c-af28-896ea269150a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:50.915244 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1b43f61-5e1d-4f3c-af28-896ea269150a service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:50.924718 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1006/2027] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:26:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 27 01:26:51.209842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d831d6e9-22eb-4543-868e-5e1fcc587e6e None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:51.211381 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-d831d6e9-22eb-4543-868e-5e1fcc587e6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:51.228831 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d831d6e9-22eb-4543-868e-5e1fcc587e6e None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:51.233413 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01071dcb-828e-451b-9f19-24270604e2d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:51.234045 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01071dcb-828e-451b-9f19-24270604e2d0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:51.234327 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01071dcb-828e-451b-9f19-24270604e2d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:51.234651 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01071dcb-828e-451b-9f19-24270604e2d0 admin admin] PATH_INFO: `/v3/projects/04bd52fbfb294ed194fb49a25c5c1ddd` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:51.235803 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01071dcb-828e-451b-9f19-24270604e2d0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=04bd52fbfb294ed194fb49a25c5c1ddd)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:51.239507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01071dcb-828e-451b-9f19-24270604e2d0 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:51.288660 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-01071dcb-828e-451b-9f19-24270604e2d0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '04bd52fbfb294ed194fb49a25c5c1ddd', 'request_id': 'req-01071dcb-828e-451b-9f19-24270604e2d0'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:51.296392 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-01071dcb-828e-451b-9f19-24270604e2d0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '04bd52fbfb294ed194fb49a25c5c1ddd', 'request_id': 'req-01071dcb-828e-451b-9f19-24270604e2d0'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:51.304021 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-01071dcb-828e-451b-9f19-24270604e2d0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '04bd52fbfb294ed194fb49a25c5c1ddd', 'request_id': 'req-01071dcb-828e-451b-9f19-24270604e2d0'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:51.314746 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-01071dcb-828e-451b-9f19-24270604e2d0 admin admin] The token cache is being invalidate because project 04bd52fbfb294ed194fb49a25c5c1ddd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:26:51.315034 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-01071dcb-828e-451b-9f19-24270604e2d0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:51.315673 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-01071dcb-828e-451b-9f19-24270604e2d0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:51.317618 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1022/2028] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:26:51 2022] DELETE /identity/v3/projects/04bd52fbfb294ed194fb49a25c5c1ddd => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:51.322164 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1b43f61-5e1d-4f3c-af28-896ea269150a service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:51.336267 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1b43f61-5e1d-4f3c-af28-896ea269150a service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:51.398056 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d5fe730-37b6-4a50-9d67-ae9feb90cab2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:51.398731 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5fe730-37b6-4a50-9d67-ae9feb90cab2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:51.399010 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5fe730-37b6-4a50-9d67-ae9feb90cab2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:51.399283 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5fe730-37b6-4a50-9d67-ae9feb90cab2 admin admin] PATH_INFO: `/v3/users/d9542cc358ab4b40b180105d5823cabf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:51.400574 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d5fe730-37b6-4a50-9d67-ae9feb90cab2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d9542cc358ab4b40b180105d5823cabf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:51.402673 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d5fe730-37b6-4a50-9d67-ae9feb90cab2 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:51.451554 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2d5fe730-37b6-4a50-9d67-ae9feb90cab2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd9542cc358ab4b40b180105d5823cabf'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:51.468562 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2d5fe730-37b6-4a50-9d67-ae9feb90cab2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd9542cc358ab4b40b180105d5823cabf'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:51.469324 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2d5fe730-37b6-4a50-9d67-ae9feb90cab2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd9542cc358ab4b40b180105d5823cabf'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:51.481869 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:INSERT=2,keystone:DELETE=13,keystone:SELECT=85 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:51.483173 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2d5fe730-37b6-4a50-9d67-ae9feb90cab2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd9542cc358ab4b40b180105d5823cabf'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:51.492011 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2d5fe730-37b6-4a50-9d67-ae9feb90cab2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd9542cc358ab4b40b180105d5823cabf'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:51.498051 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1007/2029] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:26:51 2022] DELETE /identity/v3/users/d9542cc358ab4b40b180105d5823cabf => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:51.849970 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01071dcb-828e-451b-9f19-24270604e2d0 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:51.851530 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-01071dcb-828e-451b-9f19-24270604e2d0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:51.865293 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01071dcb-828e-451b-9f19-24270604e2d0 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:51.882206 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:51.949529 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8350e2-3a22-4501-924e-5991884b4f17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:51.950182 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8350e2-3a22-4501-924e-5991884b4f17 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:51.950465 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8350e2-3a22-4501-924e-5991884b4f17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:51.950743 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8350e2-3a22-4501-924e-5991884b4f17 admin admin] PATH_INFO: `/v3/projects/c1b062a7917a4a788214b2d85f132c1e` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:51.952024 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8350e2-3a22-4501-924e-5991884b4f17 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c1b062a7917a4a788214b2d85f132c1e)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:51.953981 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8350e2-3a22-4501-924e-5991884b4f17 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:52.001516 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-7c8350e2-3a22-4501-924e-5991884b4f17 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c1b062a7917a4a788214b2d85f132c1e', 'request_id': 'req-7c8350e2-3a22-4501-924e-5991884b4f17'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:52.010400 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-7c8350e2-3a22-4501-924e-5991884b4f17 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c1b062a7917a4a788214b2d85f132c1e', 'request_id': 'req-7c8350e2-3a22-4501-924e-5991884b4f17'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:52.017261 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-7c8350e2-3a22-4501-924e-5991884b4f17 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c1b062a7917a4a788214b2d85f132c1e', 'request_id': 'req-7c8350e2-3a22-4501-924e-5991884b4f17'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:52.028252 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-7c8350e2-3a22-4501-924e-5991884b4f17 admin admin] The token cache is being invalidate because project c1b062a7917a4a788214b2d85f132c1e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:26:52.028548 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-7c8350e2-3a22-4501-924e-5991884b4f17 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:52.029256 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-7c8350e2-3a22-4501-924e-5991884b4f17 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:52.030764 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1023/2030] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:26:51 2022] DELETE /identity/v3/projects/c1b062a7917a4a788214b2d85f132c1e => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:52.035573 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92a0694-be46-458e-85c7-6bec72a2febb None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:52.035904 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92a0694-be46-458e-85c7-6bec72a2febb None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:52.036212 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92a0694-be46-458e-85c7-6bec72a2febb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:52.057196 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-a92a0694-be46-458e-85c7-6bec72a2febb None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:52.058430 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-a92a0694-be46-458e-85c7-6bec72a2febb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:52.118478 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1008/2031] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:26:52.126261 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27d54a9-aa7f-499f-bcb5-096857d6b33c None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:52.126540 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27d54a9-aa7f-499f-bcb5-096857d6b33c None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:52.126808 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27d54a9-aa7f-499f-bcb5-096857d6b33c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:52.146945 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-b27d54a9-aa7f-499f-bcb5-096857d6b33c None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:52.148134 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-b27d54a9-aa7f-499f-bcb5-096857d6b33c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:52.156323 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1024/2032] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:52.161215 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92a0694-be46-458e-85c7-6bec72a2febb None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:52.170865 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92a0694-be46-458e-85c7-6bec72a2febb None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:52.176103 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1b2882-b988-4d7c-8629-616889d1040a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:52.176688 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1b2882-b988-4d7c-8629-616889d1040a admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:52.176967 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1b2882-b988-4d7c-8629-616889d1040a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:52.177276 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1b2882-b988-4d7c-8629-616889d1040a admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:52.177920 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c1b2882-b988-4d7c-8629-616889d1040a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:26:52.178219 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c1b2882-b988-4d7c-8629-616889d1040a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:52.180054 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c1b2882-b988-4d7c-8629-616889d1040a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:52.192117 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1009/2033] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:26:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:26:52.196498 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27d54a9-aa7f-499f-bcb5-096857d6b33c None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:52.206007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27d54a9-aa7f-499f-bcb5-096857d6b33c None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:52.210614 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f412cf6-510c-45be-8f60-61afb2edd04c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:52.211203 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f412cf6-510c-45be-8f60-61afb2edd04c admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:52.211484 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f412cf6-510c-45be-8f60-61afb2edd04c admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:52.211757 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f412cf6-510c-45be-8f60-61afb2edd04c admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:52.212389 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f412cf6-510c-45be-8f60-61afb2edd04c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:52.214359 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f412cf6-510c-45be-8f60-61afb2edd04c admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:52.243667 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1025/2034] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:26:52 2022] POST /identity/v3/projects => generated 379 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:26:52.247733 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1b2882-b988-4d7c-8629-616889d1040a admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:52.258704 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1b2882-b988-4d7c-8629-616889d1040a admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:52.317173 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4c9b1f-8dd3-4aa8-b8b3-fd8d809973bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:52.317786 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4c9b1f-8dd3-4aa8-b8b3-fd8d809973bb admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:52.318067 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4c9b1f-8dd3-4aa8-b8b3-fd8d809973bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:52.318407 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4c9b1f-8dd3-4aa8-b8b3-fd8d809973bb admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:52.319321 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4c9b1f-8dd3-4aa8-b8b3-fd8d809973bb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:52.321355 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4c9b1f-8dd3-4aa8-b8b3-fd8d809973bb admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:52.342623 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1010/2035] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:26:52 2022] POST /identity/v3/users => generated 290 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:26:52.347336 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f412cf6-510c-45be-8f60-61afb2edd04c admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:52.358153 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f412cf6-510c-45be-8f60-61afb2edd04c admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:52.363775 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a70f8cf-b035-45a7-a56e-357e4968941a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:52.364272 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a70f8cf-b035-45a7-a56e-357e4968941a admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:52.364681 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a70f8cf-b035-45a7-a56e-357e4968941a admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:52.364953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a70f8cf-b035-45a7-a56e-357e4968941a admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:52.365734 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a70f8cf-b035-45a7-a56e-357e4968941a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:52.367208 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a70f8cf-b035-45a7-a56e-357e4968941a admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:52.377849 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1026/2036] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:26:52 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:26:52.381858 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4c9b1f-8dd3-4aa8-b8b3-fd8d809973bb admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:52.392242 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4c9b1f-8dd3-4aa8-b8b3-fd8d809973bb admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:52.397992 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f82a9a-199f-4de2-8595-54469c7f60cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:52.398537 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f82a9a-199f-4de2-8595-54469c7f60cb admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:52.398817 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f82a9a-199f-4de2-8595-54469c7f60cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:52.399087 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f82a9a-199f-4de2-8595-54469c7f60cb admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:52.399731 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29f82a9a-199f-4de2-8595-54469c7f60cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:52.401479 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29f82a9a-199f-4de2-8595-54469c7f60cb admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:52.414065 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1011/2037] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:26:52 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 27 01:26:52.418211 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a70f8cf-b035-45a7-a56e-357e4968941a admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:52.432165 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a70f8cf-b035-45a7-a56e-357e4968941a admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:52.437437 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ecb6f9c-02eb-4d43-bb76-07bc58ade381 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:52.438315 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ecb6f9c-02eb-4d43-bb76-07bc58ade381 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:52.438592 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ecb6f9c-02eb-4d43-bb76-07bc58ade381 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:52.438867 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ecb6f9c-02eb-4d43-bb76-07bc58ade381 admin admin] PATH_INFO: `/v3/projects/de770386fc4b4f309c038c47da3bd95f/users/a9e0c627793f40cabe0b38b925d61ecb/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:52.459479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ecb6f9c-02eb-4d43-bb76-07bc58ade381 admin admin] RBAC: Authorizing `identity:create_grant(project_id=de770386fc4b4f309c038c47da3bd95f, user_id=a9e0c627793f40cabe0b38b925d61ecb, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:52.462401 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ecb6f9c-02eb-4d43-bb76-07bc58ade381 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:52.481084 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1027/2038] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:26:52 2022] PUT /identity/v3/projects/de770386fc4b4f309c038c47da3bd95f/users/a9e0c627793f40cabe0b38b925d61ecb/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:52.492205 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33510a9d-8d38-45a0-b2d0-ca9aa763ce2a None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:52.492487 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33510a9d-8d38-45a0-b2d0-ca9aa763ce2a None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:52.492761 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33510a9d-8d38-45a0-b2d0-ca9aa763ce2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:52.528006 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-33510a9d-8d38-45a0-b2d0-ca9aa763ce2a None None] MFA Rules not processed for user `a9e0c627793f40cabe0b38b925d61ecb`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:52.529340 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-33510a9d-8d38-45a0-b2d0-ca9aa763ce2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:52.581050 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:52.611951 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1012/2039] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:52 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:52.616657 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ecb6f9c-02eb-4d43-bb76-07bc58ade381 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:52.693723 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:52.727810 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ecb6f9c-02eb-4d43-bb76-07bc58ade381 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:52.732650 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01818d2-ccb7-44f5-bc03-2d8aa223c2bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:52.733352 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01818d2-ccb7-44f5-bc03-2d8aa223c2bd admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:52.733704 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01818d2-ccb7-44f5-bc03-2d8aa223c2bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:52.734066 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01818d2-ccb7-44f5-bc03-2d8aa223c2bd admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:52.734746 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a01818d2-ccb7-44f5-bc03-2d8aa223c2bd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:52.736641 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a01818d2-ccb7-44f5-bc03-2d8aa223c2bd admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:52.767245 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1028/2040] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:26:52 2022] POST /identity/v3/projects => generated 381 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:26:52.771553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33510a9d-8d38-45a0-b2d0-ca9aa763ce2a None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:52.782218 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33510a9d-8d38-45a0-b2d0-ca9aa763ce2a None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:52.843256 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74570018-0be9-4007-9c8b-5ce02c96329a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:52.843866 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74570018-0be9-4007-9c8b-5ce02c96329a admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:52.844140 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74570018-0be9-4007-9c8b-5ce02c96329a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:52.844404 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74570018-0be9-4007-9c8b-5ce02c96329a admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:52.845038 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74570018-0be9-4007-9c8b-5ce02c96329a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:52.847134 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74570018-0be9-4007-9c8b-5ce02c96329a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:52.869132 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1013/2041] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:26:52 2022] POST /identity/v3/users => generated 297 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:26:52.873032 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01818d2-ccb7-44f5-bc03-2d8aa223c2bd admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:52.884894 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01818d2-ccb7-44f5-bc03-2d8aa223c2bd admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:52.889688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d419ba9e-cfee-4c0a-8088-a23b0ee4282b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:52.890292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d419ba9e-cfee-4c0a-8088-a23b0ee4282b admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:52.890568 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d419ba9e-cfee-4c0a-8088-a23b0ee4282b admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:52.890837 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d419ba9e-cfee-4c0a-8088-a23b0ee4282b admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:52.891469 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d419ba9e-cfee-4c0a-8088-a23b0ee4282b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:52.893052 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d419ba9e-cfee-4c0a-8088-a23b0ee4282b admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:52.904877 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1029/2042] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:26:52 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:26:52.909383 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74570018-0be9-4007-9c8b-5ce02c96329a admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:52.919828 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74570018-0be9-4007-9c8b-5ce02c96329a admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:52.925831 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1279286-db16-496c-8970-e70b2fce95fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:52.926504 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1279286-db16-496c-8970-e70b2fce95fe admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:52.926763 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1279286-db16-496c-8970-e70b2fce95fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:52.927032 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1279286-db16-496c-8970-e70b2fce95fe admin admin] PATH_INFO: `/v3/projects/c4b3ab995a99431389a52fbfdab8cfca/users/934947017d0146e089dadc547faa85ed/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:52.943734 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1279286-db16-496c-8970-e70b2fce95fe admin admin] RBAC: Authorizing `identity:create_grant(project_id=c4b3ab995a99431389a52fbfdab8cfca, user_id=934947017d0146e089dadc547faa85ed, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:52.946450 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1279286-db16-496c-8970-e70b2fce95fe admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:52.963474 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1014/2043] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:26:52 2022] PUT /identity/v3/projects/c4b3ab995a99431389a52fbfdab8cfca/users/934947017d0146e089dadc547faa85ed/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:52.967644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d419ba9e-cfee-4c0a-8088-a23b0ee4282b admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:52.977774 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d419ba9e-cfee-4c0a-8088-a23b0ee4282b admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:53.042294 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e5567b-ce9f-4968-b356-e9cd76f2196b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:53.043261 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e5567b-ce9f-4968-b356-e9cd76f2196b admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:53.043762 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e5567b-ce9f-4968-b356-e9cd76f2196b admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:53.044235 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e5567b-ce9f-4968-b356-e9cd76f2196b admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:53.045007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e5567b-ce9f-4968-b356-e9cd76f2196b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:53.046816 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e5567b-ce9f-4968-b356-e9cd76f2196b admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:53.061186 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1030/2044] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:26:52 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:26:53.062181 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1279286-db16-496c-8970-e70b2fce95fe admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:53.072870 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1279286-db16-496c-8970-e70b2fce95fe admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:53.077846 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb8a3ba-b294-4912-a9bb-2e8ec5948a95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:53.078519 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb8a3ba-b294-4912-a9bb-2e8ec5948a95 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:53.078799 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb8a3ba-b294-4912-a9bb-2e8ec5948a95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:53.079066 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb8a3ba-b294-4912-a9bb-2e8ec5948a95 admin admin] PATH_INFO: `/v3/projects/c4b3ab995a99431389a52fbfdab8cfca/users/934947017d0146e089dadc547faa85ed/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:53.081002 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb8a3ba-b294-4912-a9bb-2e8ec5948a95 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c4b3ab995a99431389a52fbfdab8cfca, user_id=934947017d0146e089dadc547faa85ed, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:53.083716 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb8a3ba-b294-4912-a9bb-2e8ec5948a95 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:53.094235 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1015/2045] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:26:53 2022] PUT /identity/v3/projects/c4b3ab995a99431389a52fbfdab8cfca/users/934947017d0146e089dadc547faa85ed/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:26:53.105962 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec76c45-7270-4166-bf0f-821933ad20f8 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:53.106245 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec76c45-7270-4166-bf0f-821933ad20f8 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:53.106517 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec76c45-7270-4166-bf0f-821933ad20f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:53.136087 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-7ec76c45-7270-4166-bf0f-821933ad20f8 None None] MFA Rules not processed for user `934947017d0146e089dadc547faa85ed`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:53.137491 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-7ec76c45-7270-4166-bf0f-821933ad20f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:53.208477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1031/2046] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:53 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:53.219202 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb8a3ba-b294-4912-a9bb-2e8ec5948a95 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:53.220905 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-1bb8a3ba-b294-4912-a9bb-2e8ec5948a95 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:53.279083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb8a3ba-b294-4912-a9bb-2e8ec5948a95 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:53.282741 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de029267-5b82-4a49-aa1d-6840c5d79933 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:53.283347 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de029267-5b82-4a49-aa1d-6840c5d79933 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:53.283624 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de029267-5b82-4a49-aa1d-6840c5d79933 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:53.283895 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de029267-5b82-4a49-aa1d-6840c5d79933 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:53.293614 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de029267-5b82-4a49-aa1d-6840c5d79933 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:53.295043 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de029267-5b82-4a49-aa1d-6840c5d79933 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:53.305212 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1016/2047] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:26:53 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:26:53.788935 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec76c45-7270-4166-bf0f-821933ad20f8 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:53.849675 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:53.874021 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec76c45-7270-4166-bf0f-821933ad20f8 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:53.881346 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca46743-c70b-48af-910f-a44acc7f9645 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:53.882093 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca46743-c70b-48af-910f-a44acc7f9645 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:53.882386 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca46743-c70b-48af-910f-a44acc7f9645 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:53.882714 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca46743-c70b-48af-910f-a44acc7f9645 admin admin] PATH_INFO: `/v3/users/a9e0c627793f40cabe0b38b925d61ecb` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:53.884632 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ca46743-c70b-48af-910f-a44acc7f9645 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a9e0c627793f40cabe0b38b925d61ecb)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:53.890199 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ca46743-c70b-48af-910f-a44acc7f9645 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:53.948324 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-4ca46743-c70b-48af-910f-a44acc7f9645 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a9e0c627793f40cabe0b38b925d61ecb'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:53.958215 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-4ca46743-c70b-48af-910f-a44acc7f9645 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a9e0c627793f40cabe0b38b925d61ecb'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:53.973619 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-4ca46743-c70b-48af-910f-a44acc7f9645 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a9e0c627793f40cabe0b38b925d61ecb'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:53.974364 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-4ca46743-c70b-48af-910f-a44acc7f9645 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a9e0c627793f40cabe0b38b925d61ecb'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:53.975485 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-4ca46743-c70b-48af-910f-a44acc7f9645 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a9e0c627793f40cabe0b38b925d61ecb'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:53.990076 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1032/2048] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:26:53 2022] DELETE /identity/v3/users/a9e0c627793f40cabe0b38b925d61ecb => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:53.994613 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0488895-7a2b-4c30-90df-7f412fdc4c37 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:53.995217 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0488895-7a2b-4c30-90df-7f412fdc4c37 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:53.995217 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0488895-7a2b-4c30-90df-7f412fdc4c37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:54.021175 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-d0488895-7a2b-4c30-90df-7f412fdc4c37 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:54.022467 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d0488895-7a2b-4c30-90df-7f412fdc4c37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:54.062569 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:54.096385 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1017/2049] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:26:54.106226 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca46743-c70b-48af-910f-a44acc7f9645 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:54.107646 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-4ca46743-c70b-48af-910f-a44acc7f9645 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:54.177521 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca46743-c70b-48af-910f-a44acc7f9645 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:54.181526 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa774f76-fc37-4ec5-9452-767b00395da9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:54.182142 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa774f76-fc37-4ec5-9452-767b00395da9 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:54.182410 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa774f76-fc37-4ec5-9452-767b00395da9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:54.182811 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa774f76-fc37-4ec5-9452-767b00395da9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:54.193145 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa774f76-fc37-4ec5-9452-767b00395da9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:54.194574 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa774f76-fc37-4ec5-9452-767b00395da9 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:54.205258 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1033/2050] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:26:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:26:54.516954 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0488895-7a2b-4c30-90df-7f412fdc4c37 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:54.518484 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d0488895-7a2b-4c30-90df-7f412fdc4c37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:54.544938 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0488895-7a2b-4c30-90df-7f412fdc4c37 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:54.564797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df78c042-2407-4449-a344-93bb86c13b72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:54.565380 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df78c042-2407-4449-a344-93bb86c13b72 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:54.565704 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df78c042-2407-4449-a344-93bb86c13b72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:54.565977 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df78c042-2407-4449-a344-93bb86c13b72 admin admin] PATH_INFO: `/v3/projects/de770386fc4b4f309c038c47da3bd95f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:54.567133 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df78c042-2407-4449-a344-93bb86c13b72 admin admin] RBAC: Authorizing `identity:delete_project(project_id=de770386fc4b4f309c038c47da3bd95f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:54.569174 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df78c042-2407-4449-a344-93bb86c13b72 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:54.629695 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-df78c042-2407-4449-a344-93bb86c13b72 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'de770386fc4b4f309c038c47da3bd95f', 'request_id': 'req-df78c042-2407-4449-a344-93bb86c13b72'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:54.638453 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-df78c042-2407-4449-a344-93bb86c13b72 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'de770386fc4b4f309c038c47da3bd95f', 'request_id': 'req-df78c042-2407-4449-a344-93bb86c13b72'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:54.645536 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-df78c042-2407-4449-a344-93bb86c13b72 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'de770386fc4b4f309c038c47da3bd95f', 'request_id': 'req-df78c042-2407-4449-a344-93bb86c13b72'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:54.660941 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-df78c042-2407-4449-a344-93bb86c13b72 admin admin] The token cache is being invalidate because project de770386fc4b4f309c038c47da3bd95f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:26:54.661260 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-df78c042-2407-4449-a344-93bb86c13b72 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:54.661924 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-df78c042-2407-4449-a344-93bb86c13b72 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:54.663476 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1018/2051] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:26:54 2022] DELETE /identity/v3/projects/de770386fc4b4f309c038c47da3bd95f => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:54.667504 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa774f76-fc37-4ec5-9452-767b00395da9 service neutron] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:54.677672 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa774f76-fc37-4ec5-9452-767b00395da9 service neutron] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:54.737160 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564dfad7-4481-4ab3-a164-05271a60e1fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:54.737781 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564dfad7-4481-4ab3-a164-05271a60e1fc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:54.738141 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564dfad7-4481-4ab3-a164-05271a60e1fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:54.738498 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564dfad7-4481-4ab3-a164-05271a60e1fc admin admin] PATH_INFO: `/v3/users/934947017d0146e089dadc547faa85ed` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:54.739700 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564dfad7-4481-4ab3-a164-05271a60e1fc admin admin] RBAC: Authorizing `identity:delete_user(user_id=934947017d0146e089dadc547faa85ed)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:54.741704 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564dfad7-4481-4ab3-a164-05271a60e1fc admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:54.744947 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:54.818662 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-564dfad7-4481-4ab3-a164-05271a60e1fc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '934947017d0146e089dadc547faa85ed'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:54.830754 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-564dfad7-4481-4ab3-a164-05271a60e1fc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '934947017d0146e089dadc547faa85ed'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:54.850748 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-564dfad7-4481-4ab3-a164-05271a60e1fc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '934947017d0146e089dadc547faa85ed'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:54.851773 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-564dfad7-4481-4ab3-a164-05271a60e1fc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '934947017d0146e089dadc547faa85ed'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:54.852771 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-564dfad7-4481-4ab3-a164-05271a60e1fc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '934947017d0146e089dadc547faa85ed'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:54.871797 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1034/2052] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:26:54 2022] DELETE /identity/v3/users/934947017d0146e089dadc547faa85ed => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:55.203534 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df78c042-2407-4449-a344-93bb86c13b72 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:55.205924 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-df78c042-2407-4449-a344-93bb86c13b72 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:55.246218 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df78c042-2407-4449-a344-93bb86c13b72 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:55.315250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50c1314-c26d-46ed-aed8-abbed20b1e61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:55.316038 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50c1314-c26d-46ed-aed8-abbed20b1e61 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:55.316546 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50c1314-c26d-46ed-aed8-abbed20b1e61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:55.317111 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50c1314-c26d-46ed-aed8-abbed20b1e61 admin admin] PATH_INFO: `/v3/projects/c4b3ab995a99431389a52fbfdab8cfca` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:55.319040 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d50c1314-c26d-46ed-aed8-abbed20b1e61 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c4b3ab995a99431389a52fbfdab8cfca)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:55.321191 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d50c1314-c26d-46ed-aed8-abbed20b1e61 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:55.373038 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d50c1314-c26d-46ed-aed8-abbed20b1e61 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c4b3ab995a99431389a52fbfdab8cfca', 'request_id': 'req-d50c1314-c26d-46ed-aed8-abbed20b1e61'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:55.382886 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d50c1314-c26d-46ed-aed8-abbed20b1e61 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c4b3ab995a99431389a52fbfdab8cfca', 'request_id': 'req-d50c1314-c26d-46ed-aed8-abbed20b1e61'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:55.390732 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:55.392415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d50c1314-c26d-46ed-aed8-abbed20b1e61 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c4b3ab995a99431389a52fbfdab8cfca', 'request_id': 'req-d50c1314-c26d-46ed-aed8-abbed20b1e61'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:55.408602 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d50c1314-c26d-46ed-aed8-abbed20b1e61 admin admin] The token cache is being invalidate because project c4b3ab995a99431389a52fbfdab8cfca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:26:55.408860 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d50c1314-c26d-46ed-aed8-abbed20b1e61 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:55.410274 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d50c1314-c26d-46ed-aed8-abbed20b1e61 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:26:55.412075 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1019/2053] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:26:55 2022] DELETE /identity/v3/projects/c4b3ab995a99431389a52fbfdab8cfca => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:55.417203 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14267ca1-1c84-4092-9708-f5d09d238201 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:55.417946 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14267ca1-1c84-4092-9708-f5d09d238201 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:55.418236 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14267ca1-1c84-4092-9708-f5d09d238201 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:55.445029 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-14267ca1-1c84-4092-9708-f5d09d238201 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:55.446403 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-14267ca1-1c84-4092-9708-f5d09d238201 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:55.524562 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1035/2054] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:55.532605 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b63736-1a4e-48d7-b82c-99bf623f1ba5 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:55.532960 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b63736-1a4e-48d7-b82c-99bf623f1ba5 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:55.533242 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b63736-1a4e-48d7-b82c-99bf623f1ba5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:55.559470 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-b4b63736-1a4e-48d7-b82c-99bf623f1ba5 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:55.570247 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-b4b63736-1a4e-48d7-b82c-99bf623f1ba5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:55.578669 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1020/2055] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:55.583209 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14267ca1-1c84-4092-9708-f5d09d238201 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:55.593833 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14267ca1-1c84-4092-9708-f5d09d238201 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:55.599801 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac5609e-f820-4b6e-9171-37162c535633 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:55.600409 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac5609e-f820-4b6e-9171-37162c535633 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:55.600687 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac5609e-f820-4b6e-9171-37162c535633 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:55.600955 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac5609e-f820-4b6e-9171-37162c535633 admin admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:55.601682 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac5609e-f820-4b6e-9171-37162c535633 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:26:55.602090 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac5609e-f820-4b6e-9171-37162c535633 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:55.608084 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac5609e-f820-4b6e-9171-37162c535633 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:55.625140 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1036/2056] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:26:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:26:55.626241 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b63736-1a4e-48d7-b82c-99bf623f1ba5 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:55.636203 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b63736-1a4e-48d7-b82c-99bf623f1ba5 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:55.641778 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0ad0a1-b4da-4010-92b9-9efaa1e4be09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:55.642401 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0ad0a1-b4da-4010-92b9-9efaa1e4be09 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:55.642680 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0ad0a1-b4da-4010-92b9-9efaa1e4be09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:55.642946 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0ad0a1-b4da-4010-92b9-9efaa1e4be09 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:55.644100 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce0ad0a1-b4da-4010-92b9-9efaa1e4be09 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:55.646045 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce0ad0a1-b4da-4010-92b9-9efaa1e4be09 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:55.674860 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1021/2057] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:26:55 2022] POST /identity/v3/projects => generated 385 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:26:55.679620 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac5609e-f820-4b6e-9171-37162c535633 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:55.691144 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac5609e-f820-4b6e-9171-37162c535633 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:55.755635 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e78676-8866-4bca-869f-c1b4b8d7b3db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:55.756254 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e78676-8866-4bca-869f-c1b4b8d7b3db admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:55.756532 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e78676-8866-4bca-869f-c1b4b8d7b3db admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:55.756805 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e78676-8866-4bca-869f-c1b4b8d7b3db admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:55.757753 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8e78676-8866-4bca-869f-c1b4b8d7b3db admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:55.759723 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8e78676-8866-4bca-869f-c1b4b8d7b3db admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:55.773077 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:55.790194 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1037/2058] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:26:55 2022] POST /identity/v3/users => generated 293 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:26:55.791034 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0ad0a1-b4da-4010-92b9-9efaa1e4be09 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:55.807380 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0ad0a1-b4da-4010-92b9-9efaa1e4be09 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:55.816548 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6457699e-8a8e-4e6b-b0d3-ffb242c00be0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:55.819635 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6457699e-8a8e-4e6b-b0d3-ffb242c00be0 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:55.819809 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6457699e-8a8e-4e6b-b0d3-ffb242c00be0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:55.820148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6457699e-8a8e-4e6b-b0d3-ffb242c00be0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:55.820875 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6457699e-8a8e-4e6b-b0d3-ffb242c00be0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:55.823772 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6457699e-8a8e-4e6b-b0d3-ffb242c00be0 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:55.836971 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1022/2059] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:26:55 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:26:55.840952 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e78676-8866-4bca-869f-c1b4b8d7b3db admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:55.852688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e78676-8866-4bca-869f-c1b4b8d7b3db admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:55.862027 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0578d81d-09e6-4677-b5e4-8e81e21a7490 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:55.863217 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0578d81d-09e6-4677-b5e4-8e81e21a7490 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:55.863629 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0578d81d-09e6-4677-b5e4-8e81e21a7490 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:55.864287 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0578d81d-09e6-4677-b5e4-8e81e21a7490 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:55.867585 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0578d81d-09e6-4677-b5e4-8e81e21a7490 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:55.867585 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0578d81d-09e6-4677-b5e4-8e81e21a7490 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:55.903428 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1038/2060] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:26:55 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:26:55.907623 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6457699e-8a8e-4e6b-b0d3-ffb242c00be0 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:55.919387 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6457699e-8a8e-4e6b-b0d3-ffb242c00be0 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:55.945508 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d91800-551f-44b2-b36c-e4a3b30852d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:55.946187 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d91800-551f-44b2-b36c-e4a3b30852d9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:55.946467 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d91800-551f-44b2-b36c-e4a3b30852d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:55.946742 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d91800-551f-44b2-b36c-e4a3b30852d9 admin admin] PATH_INFO: `/v3/projects/5d4dba952333422b997b7c9a17cfd330/users/dd36365a456a44498948ca0eb59060b2/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:55.967526 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d91800-551f-44b2-b36c-e4a3b30852d9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5d4dba952333422b997b7c9a17cfd330, user_id=dd36365a456a44498948ca0eb59060b2, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:55.970641 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d91800-551f-44b2-b36c-e4a3b30852d9 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:56.028859 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dcefb18-4739-42b0-b836-f21a9d2bb2c8 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:56.029396 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dcefb18-4739-42b0-b836-f21a9d2bb2c8 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:56.029601 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dcefb18-4739-42b0-b836-f21a9d2bb2c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:56.033362 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1023/2061] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:26:55 2022] PUT /identity/v3/projects/5d4dba952333422b997b7c9a17cfd330/users/dd36365a456a44498948ca0eb59060b2/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:26:56.081173 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-2dcefb18-4739-42b0-b836-f21a9d2bb2c8 None None] MFA Rules not processed for user `dd36365a456a44498948ca0eb59060b2`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:56.082565 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-2dcefb18-4739-42b0-b836-f21a9d2bb2c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:56.155141 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1039/2062] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:56 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:56.161808 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d91800-551f-44b2-b36c-e4a3b30852d9 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:56.272101 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d91800-551f-44b2-b36c-e4a3b30852d9 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:56.318562 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3720c5-90cb-44db-a3a6-c723f22a2e71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:56.319215 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3720c5-90cb-44db-a3a6-c723f22a2e71 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:56.319492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3720c5-90cb-44db-a3a6-c723f22a2e71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:56.319760 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3720c5-90cb-44db-a3a6-c723f22a2e71 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:56.320401 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d3720c5-90cb-44db-a3a6-c723f22a2e71 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:56.322347 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d3720c5-90cb-44db-a3a6-c723f22a2e71 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:56.414179 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dcefb18-4739-42b0-b836-f21a9d2bb2c8 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:56.417254 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1024/2063] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:26:56 2022] POST /identity/v3/projects => generated 383 bytes in 249 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:26:56.446372 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dcefb18-4739-42b0-b836-f21a9d2bb2c8 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:56.527251 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a6da2f-106d-40c6-bb6c-a18a000436da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:56.528099 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a6da2f-106d-40c6-bb6c-a18a000436da admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:56.528473 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a6da2f-106d-40c6-bb6c-a18a000436da admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:56.528840 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a6da2f-106d-40c6-bb6c-a18a000436da admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:56.529580 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23a6da2f-106d-40c6-bb6c-a18a000436da admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:56.531967 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23a6da2f-106d-40c6-bb6c-a18a000436da admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:56.562582 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1040/2064] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:26:56 2022] POST /identity/v3/users => generated 298 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:26:56.566701 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3720c5-90cb-44db-a3a6-c723f22a2e71 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:56.580173 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3720c5-90cb-44db-a3a6-c723f22a2e71 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:56.585649 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd104633-715d-498d-8c06-42e92e2c5b95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:56.586265 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd104633-715d-498d-8c06-42e92e2c5b95 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:56.586541 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd104633-715d-498d-8c06-42e92e2c5b95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:56.586812 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd104633-715d-498d-8c06-42e92e2c5b95 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:56.587466 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd104633-715d-498d-8c06-42e92e2c5b95 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:56.589087 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd104633-715d-498d-8c06-42e92e2c5b95 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:56.605791 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1025/2065] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:26:56 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 27 01:26:56.608153 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a6da2f-106d-40c6-bb6c-a18a000436da admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:56.618978 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a6da2f-106d-40c6-bb6c-a18a000436da admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:56.625352 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05dd9dfa-aef4-4615-9e63-dc58e5d62bda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:56.626168 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05dd9dfa-aef4-4615-9e63-dc58e5d62bda admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:56.626447 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05dd9dfa-aef4-4615-9e63-dc58e5d62bda admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:56.626710 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05dd9dfa-aef4-4615-9e63-dc58e5d62bda admin admin] PATH_INFO: `/v3/projects/d47958b61f344f10842789e5b211454d/users/1779a6a77f9b469b9eee2ef91c4ed44f/roles/414286927fdd44688107ff5e10235acf` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:56.650551 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05dd9dfa-aef4-4615-9e63-dc58e5d62bda admin admin] RBAC: Authorizing `identity:create_grant(project_id=d47958b61f344f10842789e5b211454d, user_id=1779a6a77f9b469b9eee2ef91c4ed44f, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:56.653338 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05dd9dfa-aef4-4615-9e63-dc58e5d62bda admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:56.670651 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1041/2066] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:26:56 2022] PUT /identity/v3/projects/d47958b61f344f10842789e5b211454d/users/1779a6a77f9b469b9eee2ef91c4ed44f/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:26:56.674648 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd104633-715d-498d-8c06-42e92e2c5b95 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:56.690151 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd104633-715d-498d-8c06-42e92e2c5b95 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:56.751235 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40524839-e09a-4ef9-91da-36f7b6415e95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:56.751852 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40524839-e09a-4ef9-91da-36f7b6415e95 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:56.752130 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40524839-e09a-4ef9-91da-36f7b6415e95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:56.752433 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40524839-e09a-4ef9-91da-36f7b6415e95 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:56.753622 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40524839-e09a-4ef9-91da-36f7b6415e95 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:56.755209 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40524839-e09a-4ef9-91da-36f7b6415e95 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:56.757209 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:56.770524 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1026/2067] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:26:56 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:26:56.771734 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05dd9dfa-aef4-4615-9e63-dc58e5d62bda admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:56.782016 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05dd9dfa-aef4-4615-9e63-dc58e5d62bda admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:56.786860 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6847e8b8-2a16-49f1-9931-a3759dc0c028 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:56.787504 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6847e8b8-2a16-49f1-9931-a3759dc0c028 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:56.787780 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6847e8b8-2a16-49f1-9931-a3759dc0c028 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:56.788050 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6847e8b8-2a16-49f1-9931-a3759dc0c028 admin admin] PATH_INFO: `/v3/projects/d47958b61f344f10842789e5b211454d/users/1779a6a77f9b469b9eee2ef91c4ed44f/roles/414286927fdd44688107ff5e10235acf` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:56.790008 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6847e8b8-2a16-49f1-9931-a3759dc0c028 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d47958b61f344f10842789e5b211454d, user_id=1779a6a77f9b469b9eee2ef91c4ed44f, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:56.792653 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6847e8b8-2a16-49f1-9931-a3759dc0c028 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:56.803251 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1042/2068] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:26:56 2022] PUT /identity/v3/projects/d47958b61f344f10842789e5b211454d/users/1779a6a77f9b469b9eee2ef91c4ed44f/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:26:56.814501 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8370a1ba-c619-495a-991d-c23fd5dfe19e None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:56.814784 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8370a1ba-c619-495a-991d-c23fd5dfe19e None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:56.815061 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8370a1ba-c619-495a-991d-c23fd5dfe19e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:56.845027 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-8370a1ba-c619-495a-991d-c23fd5dfe19e None None] MFA Rules not processed for user `1779a6a77f9b469b9eee2ef91c4ed44f`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:26:56.846295 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-8370a1ba-c619-495a-991d-c23fd5dfe19e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:56.915962 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1027/2069] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:26:56 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:26:56.926907 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6847e8b8-2a16-49f1-9931-a3759dc0c028 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:26:56.928685 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-6847e8b8-2a16-49f1-9931-a3759dc0c028 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:26:56.966366 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:26:56.997192 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6847e8b8-2a16-49f1-9931-a3759dc0c028 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:26:57.000836 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ce2071-ccd9-429f-bf8c-670fd4a84462 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:26:57.001472 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ce2071-ccd9-429f-bf8c-670fd4a84462 service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:26:57.001753 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ce2071-ccd9-429f-bf8c-670fd4a84462 service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:26:57.002023 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ce2071-ccd9-429f-bf8c-670fd4a84462 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:26:57.063345 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ce2071-ccd9-429f-bf8c-670fd4a84462 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:26:57.065269 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ce2071-ccd9-429f-bf8c-670fd4a84462 service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:26:57.077153 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1043/2070] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:26:56 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:27:02.635542 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8370a1ba-c619-495a-991d-c23fd5dfe19e None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:02.637161 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-8370a1ba-c619-495a-991d-c23fd5dfe19e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:27:02.654412 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8370a1ba-c619-495a-991d-c23fd5dfe19e None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:02.658460 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa1a08b-d7d9-4129-9ccb-8cec4faddb4b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:02.659678 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa1a08b-d7d9-4129-9ccb-8cec4faddb4b service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:02.659960 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa1a08b-d7d9-4129-9ccb-8cec4faddb4b service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:02.660232 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa1a08b-d7d9-4129-9ccb-8cec4faddb4b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:02.671266 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa1a08b-d7d9-4129-9ccb-8cec4faddb4b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:02.672718 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa1a08b-d7d9-4129-9ccb-8cec4faddb4b service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:02.683620 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1028/2071] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:27:02 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:27:07.075759 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=36 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:27:12.679347 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=41 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:27:14.806171 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e48d922-c27f-46f6-8bb6-160d80167003 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:14.806708 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e48d922-c27f-46f6-8bb6-160d80167003 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:14.806925 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e48d922-c27f-46f6-8bb6-160d80167003 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:14.830249 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-2e48d922-c27f-46f6-8bb6-160d80167003 None None] MFA Rules not processed for user `7a3f8b5eb09f4ba29f4bd343bd91f6a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:27:14.831886 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-2e48d922-c27f-46f6-8bb6-160d80167003 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:27:14.891242 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1044/2072] 10.222.0.42 () {58 vars in 1009 bytes} [Thu Oct 27 01:27:14 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:27:14.947348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa1a08b-d7d9-4129-9ccb-8cec4faddb4b service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:14.948738 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-eaa1a08b-d7d9-4129-9ccb-8cec4faddb4b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:27:15.020996 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa1a08b-d7d9-4129-9ccb-8cec4faddb4b service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:15.026315 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a17a68-8867-4d77-b071-e3cbdb625119 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77f2cf9e0c3541f9bccfb7e213d88609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:15.026924 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a17a68-8867-4d77-b071-e3cbdb625119 service swift] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:15.027207 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a17a68-8867-4d77-b071-e3cbdb625119 service swift] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:15.027503 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a17a68-8867-4d77-b071-e3cbdb625119 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:15.037206 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a17a68-8867-4d77-b071-e3cbdb625119 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:15.038918 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a17a68-8867-4d77-b071-e3cbdb625119 service swift] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:15.050267 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1029/2073] 10.222.0.42 () {60 vars in 1384 bytes} [Thu Oct 27 01:27:14 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 27 01:27:17.323119 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e48d922-c27f-46f6-8bb6-160d80167003 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:17.405640 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e48d922-c27f-46f6-8bb6-160d80167003 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:17.411121 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d35048-42c2-4188-9544-5ff80867f9fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:17.411819 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d35048-42c2-4188-9544-5ff80867f9fe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:17.412045 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d35048-42c2-4188-9544-5ff80867f9fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:17.412317 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d35048-42c2-4188-9544-5ff80867f9fe admin admin] PATH_INFO: `/v3/users/dd36365a456a44498948ca0eb59060b2` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:17.413475 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7d35048-42c2-4188-9544-5ff80867f9fe admin admin] RBAC: Authorizing `identity:delete_user(user_id=dd36365a456a44498948ca0eb59060b2)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:17.415747 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7d35048-42c2-4188-9544-5ff80867f9fe admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:17.471505 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-f7d35048-42c2-4188-9544-5ff80867f9fe admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dd36365a456a44498948ca0eb59060b2'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:17.481014 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-f7d35048-42c2-4188-9544-5ff80867f9fe admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dd36365a456a44498948ca0eb59060b2'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:17.495447 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-f7d35048-42c2-4188-9544-5ff80867f9fe admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dd36365a456a44498948ca0eb59060b2'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:17.496277 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-f7d35048-42c2-4188-9544-5ff80867f9fe admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dd36365a456a44498948ca0eb59060b2'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:17.497303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-f7d35048-42c2-4188-9544-5ff80867f9fe admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dd36365a456a44498948ca0eb59060b2'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:17.512714 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1045/2074] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:27:17 2022] DELETE /identity/v3/users/dd36365a456a44498948ca0eb59060b2 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:27:17.518013 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8609b4-fd17-449f-8465-bb4d22ed4e70 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:17.518568 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8609b4-fd17-449f-8465-bb4d22ed4e70 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:17.521096 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8609b4-fd17-449f-8465-bb4d22ed4e70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:17.565809 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-dd8609b4-fd17-449f-8465-bb4d22ed4e70 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:27:17.568461 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-dd8609b4-fd17-449f-8465-bb4d22ed4e70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:27:17.655990 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1030/2075] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:27:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:27:17.666472 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d35048-42c2-4188-9544-5ff80867f9fe admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:17.668126 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-f7d35048-42c2-4188-9544-5ff80867f9fe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:27:17.705211 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:27:17.749948 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d35048-42c2-4188-9544-5ff80867f9fe admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:17.754005 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea8f887-46c1-42e7-b164-6171d7640322 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:17.754743 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea8f887-46c1-42e7-b164-6171d7640322 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:17.755121 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea8f887-46c1-42e7-b164-6171d7640322 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:17.755535 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea8f887-46c1-42e7-b164-6171d7640322 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:17.776417 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ea8f887-46c1-42e7-b164-6171d7640322 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:17.778716 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ea8f887-46c1-42e7-b164-6171d7640322 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:17.795348 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1046/2076] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:27:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 27 01:27:18.152330 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8609b4-fd17-449f-8465-bb4d22ed4e70 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:18.154202 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-dd8609b4-fd17-449f-8465-bb4d22ed4e70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:27:18.167816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8609b4-fd17-449f-8465-bb4d22ed4e70 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:18.179845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822bae21-ae50-4a30-beb9-4f4bf0fa5eeb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:18.180430 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822bae21-ae50-4a30-beb9-4f4bf0fa5eeb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:18.180711 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822bae21-ae50-4a30-beb9-4f4bf0fa5eeb admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:18.180983 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822bae21-ae50-4a30-beb9-4f4bf0fa5eeb admin admin] PATH_INFO: `/v3/projects/5d4dba952333422b997b7c9a17cfd330` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:18.182404 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-822bae21-ae50-4a30-beb9-4f4bf0fa5eeb admin admin] RBAC: Authorizing `identity:delete_project(project_id=5d4dba952333422b997b7c9a17cfd330)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:18.184332 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-822bae21-ae50-4a30-beb9-4f4bf0fa5eeb admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:18.243872 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-822bae21-ae50-4a30-beb9-4f4bf0fa5eeb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5d4dba952333422b997b7c9a17cfd330', 'request_id': 'req-822bae21-ae50-4a30-beb9-4f4bf0fa5eeb'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:18.252737 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-822bae21-ae50-4a30-beb9-4f4bf0fa5eeb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5d4dba952333422b997b7c9a17cfd330', 'request_id': 'req-822bae21-ae50-4a30-beb9-4f4bf0fa5eeb'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:18.260525 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-822bae21-ae50-4a30-beb9-4f4bf0fa5eeb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5d4dba952333422b997b7c9a17cfd330', 'request_id': 'req-822bae21-ae50-4a30-beb9-4f4bf0fa5eeb'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:18.282037 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-822bae21-ae50-4a30-beb9-4f4bf0fa5eeb admin admin] The token cache is being invalidate because project 5d4dba952333422b997b7c9a17cfd330 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:27:18.282166 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-822bae21-ae50-4a30-beb9-4f4bf0fa5eeb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:18.282890 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-822bae21-ae50-4a30-beb9-4f4bf0fa5eeb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:18.287709 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1031/2077] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:27:18 2022] DELETE /identity/v3/projects/5d4dba952333422b997b7c9a17cfd330 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:27:18.288874 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea8f887-46c1-42e7-b164-6171d7640322 service neutron] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:18.302447 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea8f887-46c1-42e7-b164-6171d7640322 service neutron] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:18.383015 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1cec46-004f-400f-988b-5d02917f1802 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:18.383784 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1cec46-004f-400f-988b-5d02917f1802 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:18.384147 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1cec46-004f-400f-988b-5d02917f1802 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:18.384421 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1cec46-004f-400f-988b-5d02917f1802 admin admin] PATH_INFO: `/v3/users/1779a6a77f9b469b9eee2ef91c4ed44f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:18.385634 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1cec46-004f-400f-988b-5d02917f1802 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1779a6a77f9b469b9eee2ef91c4ed44f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:18.387565 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1cec46-004f-400f-988b-5d02917f1802 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:18.459904 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-fd1cec46-004f-400f-988b-5d02917f1802 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1779a6a77f9b469b9eee2ef91c4ed44f'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:18.474631 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-fd1cec46-004f-400f-988b-5d02917f1802 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1779a6a77f9b469b9eee2ef91c4ed44f'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:18.501511 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-fd1cec46-004f-400f-988b-5d02917f1802 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1779a6a77f9b469b9eee2ef91c4ed44f'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:18.501511 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-fd1cec46-004f-400f-988b-5d02917f1802 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1779a6a77f9b469b9eee2ef91c4ed44f'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:18.501511 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-fd1cec46-004f-400f-988b-5d02917f1802 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1779a6a77f9b469b9eee2ef91c4ed44f'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:18.526748 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1047/2078] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:27:18 2022] DELETE /identity/v3/users/1779a6a77f9b469b9eee2ef91c4ed44f => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:27:18.956504 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822bae21-ae50-4a30-beb9-4f4bf0fa5eeb admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:18.959326 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-822bae21-ae50-4a30-beb9-4f4bf0fa5eeb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:27:18.981768 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822bae21-ae50-4a30-beb9-4f4bf0fa5eeb admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:19.002231 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:27:19.076775 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d92739-bcd4-4ccd-91c2-7849f2d87dea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:19.077473 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d92739-bcd4-4ccd-91c2-7849f2d87dea admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:19.077801 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d92739-bcd4-4ccd-91c2-7849f2d87dea admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:19.078083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d92739-bcd4-4ccd-91c2-7849f2d87dea admin admin] PATH_INFO: `/v3/projects/d47958b61f344f10842789e5b211454d` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:19.079249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d92739-bcd4-4ccd-91c2-7849f2d87dea admin admin] RBAC: Authorizing `identity:delete_project(project_id=d47958b61f344f10842789e5b211454d)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:19.081725 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d92739-bcd4-4ccd-91c2-7849f2d87dea admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:19.157636 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-e4d92739-bcd4-4ccd-91c2-7849f2d87dea admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd47958b61f344f10842789e5b211454d', 'request_id': 'req-e4d92739-bcd4-4ccd-91c2-7849f2d87dea'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:19.170033 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-e4d92739-bcd4-4ccd-91c2-7849f2d87dea admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd47958b61f344f10842789e5b211454d', 'request_id': 'req-e4d92739-bcd4-4ccd-91c2-7849f2d87dea'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:19.180724 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-e4d92739-bcd4-4ccd-91c2-7849f2d87dea admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd47958b61f344f10842789e5b211454d', 'request_id': 'req-e4d92739-bcd4-4ccd-91c2-7849f2d87dea'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:19.193283 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-e4d92739-bcd4-4ccd-91c2-7849f2d87dea admin admin] The token cache is being invalidate because project d47958b61f344f10842789e5b211454d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:27:19.193396 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-e4d92739-bcd4-4ccd-91c2-7849f2d87dea admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:19.194281 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-e4d92739-bcd4-4ccd-91c2-7849f2d87dea admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:19.197844 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1032/2079] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:27:18 2022] DELETE /identity/v3/projects/d47958b61f344f10842789e5b211454d => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:27:19.204382 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-262cedad-7079-4ab4-8e86-efd6afd31da9 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:19.204661 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-262cedad-7079-4ab4-8e86-efd6afd31da9 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:19.204929 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-262cedad-7079-4ab4-8e86-efd6afd31da9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:19.240766 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-262cedad-7079-4ab4-8e86-efd6afd31da9 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:27:19.242126 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-262cedad-7079-4ab4-8e86-efd6afd31da9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:27:19.291228 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:27:19.313168 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1048/2080] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:27:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:27:19.322186 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4b32d4-e04f-4f18-b65c-b04164abd93f None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:19.322890 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4b32d4-e04f-4f18-b65c-b04164abd93f None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:19.323164 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4b32d4-e04f-4f18-b65c-b04164abd93f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:19.344440 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-af4b32d4-e04f-4f18-b65c-b04164abd93f None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:27:19.345727 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-af4b32d4-e04f-4f18-b65c-b04164abd93f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:27:19.355427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1033/2081] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:27:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:27:19.359950 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-262cedad-7079-4ab4-8e86-efd6afd31da9 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:19.372022 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-262cedad-7079-4ab4-8e86-efd6afd31da9 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:19.379338 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66447d5c-25fa-4df2-9923-09f52fc54731 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:19.379841 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66447d5c-25fa-4df2-9923-09f52fc54731 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:19.380157 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66447d5c-25fa-4df2-9923-09f52fc54731 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:19.380382 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66447d5c-25fa-4df2-9923-09f52fc54731 admin admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:19.382180 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66447d5c-25fa-4df2-9923-09f52fc54731 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:27:19.382846 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66447d5c-25fa-4df2-9923-09f52fc54731 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:19.384494 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66447d5c-25fa-4df2-9923-09f52fc54731 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:19.397720 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1049/2082] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:27:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:27:19.402223 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af4b32d4-e04f-4f18-b65c-b04164abd93f None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:19.411675 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af4b32d4-e04f-4f18-b65c-b04164abd93f None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:19.416278 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e5e102-4f00-43e9-91af-6df37829f6b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:19.416866 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e5e102-4f00-43e9-91af-6df37829f6b2 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:19.417791 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e5e102-4f00-43e9-91af-6df37829f6b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:19.418070 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e5e102-4f00-43e9-91af-6df37829f6b2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:19.418715 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e5e102-4f00-43e9-91af-6df37829f6b2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:19.420608 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e5e102-4f00-43e9-91af-6df37829f6b2 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:19.448960 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1034/2083] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:27:19 2022] POST /identity/v3/projects => generated 385 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:27:19.453805 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66447d5c-25fa-4df2-9923-09f52fc54731 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:19.465037 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66447d5c-25fa-4df2-9923-09f52fc54731 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:19.535822 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8cf14a-b3a1-42c4-98f3-3b5f6fae1913 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:19.536330 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8cf14a-b3a1-42c4-98f3-3b5f6fae1913 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:19.536769 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8cf14a-b3a1-42c4-98f3-3b5f6fae1913 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:19.537069 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8cf14a-b3a1-42c4-98f3-3b5f6fae1913 admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:19.537743 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f8cf14a-b3a1-42c4-98f3-3b5f6fae1913 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:19.539773 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f8cf14a-b3a1-42c4-98f3-3b5f6fae1913 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:19.567745 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1050/2084] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:27:19 2022] POST /identity/v3/users => generated 293 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:27:19.571989 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e5e102-4f00-43e9-91af-6df37829f6b2 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:19.586719 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e5e102-4f00-43e9-91af-6df37829f6b2 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:19.591791 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb6390c0-3af2-4ade-b535-8e67717955d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:19.592500 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6390c0-3af2-4ade-b535-8e67717955d9 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:19.592825 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6390c0-3af2-4ade-b535-8e67717955d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:19.593203 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6390c0-3af2-4ade-b535-8e67717955d9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:19.593905 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb6390c0-3af2-4ade-b535-8e67717955d9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:19.595886 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb6390c0-3af2-4ade-b535-8e67717955d9 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:19.621210 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1035/2085] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:27:19 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:27:19.622217 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8cf14a-b3a1-42c4-98f3-3b5f6fae1913 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:19.634609 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8cf14a-b3a1-42c4-98f3-3b5f6fae1913 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:19.639514 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fc810d-f88b-453c-b37c-271570a57492 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:19.640213 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fc810d-f88b-453c-b37c-271570a57492 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:19.640555 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fc810d-f88b-453c-b37c-271570a57492 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:19.640897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fc810d-f88b-453c-b37c-271570a57492 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:19.641912 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4fc810d-f88b-453c-b37c-271570a57492 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:19.643588 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4fc810d-f88b-453c-b37c-271570a57492 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:19.665308 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1051/2086] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:27:19 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:27:19.669341 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb6390c0-3af2-4ade-b535-8e67717955d9 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:19.681242 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb6390c0-3af2-4ade-b535-8e67717955d9 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:19.686534 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf55fc6-c84d-4068-8902-0e55a7048df1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:19.687456 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf55fc6-c84d-4068-8902-0e55a7048df1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:19.687951 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf55fc6-c84d-4068-8902-0e55a7048df1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:19.688406 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf55fc6-c84d-4068-8902-0e55a7048df1 admin admin] PATH_INFO: `/v3/projects/d72ae16023c941a0ab5bc372859b81eb/users/9c4406fa56f14b6b847872ed150c3b2b/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:19.708534 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf55fc6-c84d-4068-8902-0e55a7048df1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d72ae16023c941a0ab5bc372859b81eb, user_id=9c4406fa56f14b6b847872ed150c3b2b, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:19.711799 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf55fc6-c84d-4068-8902-0e55a7048df1 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:19.729837 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1036/2087] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:27:19 2022] PUT /identity/v3/projects/d72ae16023c941a0ab5bc372859b81eb/users/9c4406fa56f14b6b847872ed150c3b2b/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:27:19.809840 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2c07ad-8791-4ee4-8ac0-a4856e284676 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:19.810194 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2c07ad-8791-4ee4-8ac0-a4856e284676 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:19.810418 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2c07ad-8791-4ee4-8ac0-a4856e284676 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:19.847486 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-7a2c07ad-8791-4ee4-8ac0-a4856e284676 None None] MFA Rules not processed for user `9c4406fa56f14b6b847872ed150c3b2b`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:27:19.848871 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-7a2c07ad-8791-4ee4-8ac0-a4856e284676 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:27:19.937217 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1052/2088] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:27:19 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:27:19.940178 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf55fc6-c84d-4068-8902-0e55a7048df1 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:20.007502 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:27:20.015732 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2c07ad-8791-4ee4-8ac0-a4856e284676 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:20.018092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-7a2c07ad-8791-4ee4-8ac0-a4856e284676 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:27:20.033106 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf55fc6-c84d-4068-8902-0e55a7048df1 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:20.043127 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d07674-6d5b-4ca3-aaec-2d3aecc223d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:20.043904 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d07674-6d5b-4ca3-aaec-2d3aecc223d0 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:20.044241 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d07674-6d5b-4ca3-aaec-2d3aecc223d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:20.044576 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d07674-6d5b-4ca3-aaec-2d3aecc223d0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:20.045340 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d07674-6d5b-4ca3-aaec-2d3aecc223d0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:20.047317 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d07674-6d5b-4ca3-aaec-2d3aecc223d0 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:20.079494 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:27:20.086650 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1037/2089] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:27:19 2022] POST /identity/v3/projects => generated 383 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:27:20.087840 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d07674-6d5b-4ca3-aaec-2d3aecc223d0 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:20.094394 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2c07ad-8791-4ee4-8ac0-a4856e284676 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:20.154880 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6851f7b2-38a9-4658-b42e-ca3334100771 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:20.155519 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6851f7b2-38a9-4658-b42e-ca3334100771 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:20.156084 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6851f7b2-38a9-4658-b42e-ca3334100771 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:20.156480 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6851f7b2-38a9-4658-b42e-ca3334100771 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:20.158425 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-6851f7b2-38a9-4658-b42e-ca3334100771 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:27:20.194655 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d07674-6d5b-4ca3-aaec-2d3aecc223d0 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:20.207367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ee96ad-d0dc-4b42-a238-af429a240a1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:20.208063 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ee96ad-d0dc-4b42-a238-af429a240a1a admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:20.208343 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ee96ad-d0dc-4b42-a238-af429a240a1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:20.208924 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ee96ad-d0dc-4b42-a238-af429a240a1a admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:20.209641 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ee96ad-d0dc-4b42-a238-af429a240a1a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:20.211159 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ee96ad-d0dc-4b42-a238-af429a240a1a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:20.240149 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1038/2090] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:27:20 2022] POST /identity/v3/users => generated 292 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:27:20.244130 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ee96ad-d0dc-4b42-a238-af429a240a1a admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:20.269324 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ee96ad-d0dc-4b42-a238-af429a240a1a admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:20.273916 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc85644-bc36-486b-b1a2-11304d26c032 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:20.274611 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc85644-bc36-486b-b1a2-11304d26c032 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:20.274949 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc85644-bc36-486b-b1a2-11304d26c032 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:20.275290 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc85644-bc36-486b-b1a2-11304d26c032 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:20.276007 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfc85644-bc36-486b-b1a2-11304d26c032 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:20.277738 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfc85644-bc36-486b-b1a2-11304d26c032 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:20.283761 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6851f7b2-38a9-4658-b42e-ca3334100771 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:20.288015 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6851f7b2-38a9-4658-b42e-ca3334100771 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:20.292650 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1039/2091] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:27:20 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:27:20.294438 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc85644-bc36-486b-b1a2-11304d26c032 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:20.304679 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1053/2092] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:27:20 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 289 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 27 01:27:20.317412 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc85644-bc36-486b-b1a2-11304d26c032 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:20.322849 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a90f37-77de-404f-9414-0ca0412f8f7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:20.323461 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a90f37-77de-404f-9414-0ca0412f8f7d admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:20.323732 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a90f37-77de-404f-9414-0ca0412f8f7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:20.323994 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a90f37-77de-404f-9414-0ca0412f8f7d admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:20.324792 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a90f37-77de-404f-9414-0ca0412f8f7d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:20.326411 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a90f37-77de-404f-9414-0ca0412f8f7d admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:20.340214 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1040/2093] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:27:20 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:27:20.344255 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6851f7b2-38a9-4658-b42e-ca3334100771 service neutron] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:20.362676 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6851f7b2-38a9-4658-b42e-ca3334100771 service neutron] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:20.369397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926f76b3-dc59-4ca2-a55d-ed8550893315 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:20.370315 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926f76b3-dc59-4ca2-a55d-ed8550893315 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:20.370586 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926f76b3-dc59-4ca2-a55d-ed8550893315 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:20.370859 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926f76b3-dc59-4ca2-a55d-ed8550893315 admin admin] PATH_INFO: `/v3/projects/9667cf28520149a5b0fcaae38c380a7a/users/4b15ee275b134dcebb43ea15cb790214/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:20.413226 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-926f76b3-dc59-4ca2-a55d-ed8550893315 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9667cf28520149a5b0fcaae38c380a7a, user_id=4b15ee275b134dcebb43ea15cb790214, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:20.416939 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-926f76b3-dc59-4ca2-a55d-ed8550893315 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:20.453809 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1054/2094] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:27:20 2022] PUT /identity/v3/projects/9667cf28520149a5b0fcaae38c380a7a/users/4b15ee275b134dcebb43ea15cb790214/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:27:20.466076 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfaae54d-2d68-4434-969f-838f521b307c None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:20.466626 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfaae54d-2d68-4434-969f-838f521b307c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:20.467104 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfaae54d-2d68-4434-969f-838f521b307c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:20.521328 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-dfaae54d-2d68-4434-969f-838f521b307c None None] MFA Rules not processed for user `4b15ee275b134dcebb43ea15cb790214`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:27:20.523165 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-dfaae54d-2d68-4434-969f-838f521b307c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:27:20.625274 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1041/2095] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:27:20 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:27:20.631543 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926f76b3-dc59-4ca2-a55d-ed8550893315 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:20.725564 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926f76b3-dc59-4ca2-a55d-ed8550893315 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:20.732059 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b71e0af-79f3-4fdf-839e-6382ea59d7f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:20.732629 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b71e0af-79f3-4fdf-839e-6382ea59d7f5 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:20.733206 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b71e0af-79f3-4fdf-839e-6382ea59d7f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:20.733751 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b71e0af-79f3-4fdf-839e-6382ea59d7f5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:20.735196 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b71e0af-79f3-4fdf-839e-6382ea59d7f5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:20.739219 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b71e0af-79f3-4fdf-839e-6382ea59d7f5 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:20.779560 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1055/2096] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:27:20 2022] POST /identity/v3/projects => generated 385 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:27:20.781262 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfaae54d-2d68-4434-969f-838f521b307c None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:20.794271 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfaae54d-2d68-4434-969f-838f521b307c None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:20.829441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:27:20.888810 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba6a1bb-8efd-4135-8df7-28af5353991a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:20.889496 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba6a1bb-8efd-4135-8df7-28af5353991a admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:20.889935 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba6a1bb-8efd-4135-8df7-28af5353991a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:20.890223 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba6a1bb-8efd-4135-8df7-28af5353991a admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:20.890875 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba6a1bb-8efd-4135-8df7-28af5353991a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:20.892845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba6a1bb-8efd-4135-8df7-28af5353991a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:20.931113 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1042/2097] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:27:20 2022] POST /identity/v3/users => generated 299 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:27:20.932829 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b71e0af-79f3-4fdf-839e-6382ea59d7f5 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:20.947007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b71e0af-79f3-4fdf-839e-6382ea59d7f5 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:20.957461 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5bcf46-e48d-4744-ba1b-715205105411 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:20.957952 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5bcf46-e48d-4744-ba1b-715205105411 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:20.958257 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5bcf46-e48d-4744-ba1b-715205105411 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:20.958531 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5bcf46-e48d-4744-ba1b-715205105411 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:20.959548 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5bcf46-e48d-4744-ba1b-715205105411 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:20.962271 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5bcf46-e48d-4744-ba1b-715205105411 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:20.965578 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:27:20.984706 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1056/2098] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:27:20 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:27:20.985946 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba6a1bb-8efd-4135-8df7-28af5353991a admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:20.999912 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba6a1bb-8efd-4135-8df7-28af5353991a admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:21.006130 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceabb31a-0069-4c6d-a765-3600899f070c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:21.006928 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceabb31a-0069-4c6d-a765-3600899f070c admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:21.007286 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceabb31a-0069-4c6d-a765-3600899f070c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:21.007637 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceabb31a-0069-4c6d-a765-3600899f070c admin admin] PATH_INFO: `/v3/projects/2e2b5ad2ee7c42f1823b3b99707c9d89/users/459069db230b4072898de596db774996/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:21.028939 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceabb31a-0069-4c6d-a765-3600899f070c admin admin] RBAC: Authorizing `identity:create_grant(project_id=2e2b5ad2ee7c42f1823b3b99707c9d89, user_id=459069db230b4072898de596db774996, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:21.031995 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceabb31a-0069-4c6d-a765-3600899f070c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:21.055046 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1043/2099] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:27:20 2022] PUT /identity/v3/projects/2e2b5ad2ee7c42f1823b3b99707c9d89/users/459069db230b4072898de596db774996/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:27:21.059209 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5bcf46-e48d-4744-ba1b-715205105411 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:21.072441 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5bcf46-e48d-4744-ba1b-715205105411 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:21.163373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b92e5d8-e53d-4601-9566-134ae0733880 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:21.164007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b92e5d8-e53d-4601-9566-134ae0733880 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:21.164291 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b92e5d8-e53d-4601-9566-134ae0733880 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:21.164607 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b92e5d8-e53d-4601-9566-134ae0733880 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:21.165409 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b92e5d8-e53d-4601-9566-134ae0733880 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:21.167048 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b92e5d8-e53d-4601-9566-134ae0733880 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:21.182931 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1057/2100] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:27:21 2022] GET /identity/v3/roles => generated 1420 bytes in 121 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:27:21.184378 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceabb31a-0069-4c6d-a765-3600899f070c admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:21.197335 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceabb31a-0069-4c6d-a765-3600899f070c admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:21.202854 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be7e8dd-ea67-4cc4-9fce-494275477c45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:21.203828 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be7e8dd-ea67-4cc4-9fce-494275477c45 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:21.204275 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be7e8dd-ea67-4cc4-9fce-494275477c45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:21.204726 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be7e8dd-ea67-4cc4-9fce-494275477c45 admin admin] PATH_INFO: `/v3/projects/2e2b5ad2ee7c42f1823b3b99707c9d89/users/459069db230b4072898de596db774996/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:21.207807 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be7e8dd-ea67-4cc4-9fce-494275477c45 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2e2b5ad2ee7c42f1823b3b99707c9d89, user_id=459069db230b4072898de596db774996, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:21.210976 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be7e8dd-ea67-4cc4-9fce-494275477c45 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:21.232463 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1044/2101] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:27:21 2022] PUT /identity/v3/projects/2e2b5ad2ee7c42f1823b3b99707c9d89/users/459069db230b4072898de596db774996/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:27:21.247961 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e886c3-9ffd-406a-9588-96004449a5a1 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:21.248455 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e886c3-9ffd-406a-9588-96004449a5a1 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:21.248901 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e886c3-9ffd-406a-9588-96004449a5a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:21.286843 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-02e886c3-9ffd-406a-9588-96004449a5a1 None None] MFA Rules not processed for user `459069db230b4072898de596db774996`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:27:21.288631 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-02e886c3-9ffd-406a-9588-96004449a5a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:27:21.402510 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1058/2102] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:27:21 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:27:21.416169 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be7e8dd-ea67-4cc4-9fce-494275477c45 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:21.420111 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-9be7e8dd-ea67-4cc4-9fce-494275477c45 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:27:21.497138 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be7e8dd-ea67-4cc4-9fce-494275477c45 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:21.501442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3756de25-7469-4e1d-a666-45e2510fb2b3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:21.502202 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3756de25-7469-4e1d-a666-45e2510fb2b3 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:21.502619 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3756de25-7469-4e1d-a666-45e2510fb2b3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:21.502966 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3756de25-7469-4e1d-a666-45e2510fb2b3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:21.604941 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3756de25-7469-4e1d-a666-45e2510fb2b3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:21.606880 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3756de25-7469-4e1d-a666-45e2510fb2b3 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:21.624804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1045/2103] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:27:21 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 209 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:27:23.479628 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e886c3-9ffd-406a-9588-96004449a5a1 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:23.492511 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e886c3-9ffd-406a-9588-96004449a5a1 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:23.498029 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e7d499-55f7-4726-8cf2-bd4076a0539a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:23.499057 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e7d499-55f7-4726-8cf2-bd4076a0539a service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:23.499549 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e7d499-55f7-4726-8cf2-bd4076a0539a service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:23.500524 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e7d499-55f7-4726-8cf2-bd4076a0539a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:23.577451 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25e7d499-55f7-4726-8cf2-bd4076a0539a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:23.579400 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25e7d499-55f7-4726-8cf2-bd4076a0539a service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:23.594110 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1059/2104] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:27:23 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:27:23.964309 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3756de25-7469-4e1d-a666-45e2510fb2b3 service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:23.966492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-3756de25-7469-4e1d-a666-45e2510fb2b3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:27:24.014879 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:27:24.095952 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3756de25-7469-4e1d-a666-45e2510fb2b3 service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:24.111000 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4562dc7e-6217-49bd-9e85-f227e2e59a46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:24.112323 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4562dc7e-6217-49bd-9e85-f227e2e59a46 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:24.112841 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4562dc7e-6217-49bd-9e85-f227e2e59a46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:24.113983 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4562dc7e-6217-49bd-9e85-f227e2e59a46 admin admin] PATH_INFO: `/v3/users/0ca2f266188f4001abf68776b42f2f82` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:24.115792 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4562dc7e-6217-49bd-9e85-f227e2e59a46 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0ca2f266188f4001abf68776b42f2f82)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:24.118592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4562dc7e-6217-49bd-9e85-f227e2e59a46 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:24.200977 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4562dc7e-6217-49bd-9e85-f227e2e59a46 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0ca2f266188f4001abf68776b42f2f82'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:24.216475 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4562dc7e-6217-49bd-9e85-f227e2e59a46 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0ca2f266188f4001abf68776b42f2f82'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:24.217556 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4562dc7e-6217-49bd-9e85-f227e2e59a46 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0ca2f266188f4001abf68776b42f2f82'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:24.237352 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4562dc7e-6217-49bd-9e85-f227e2e59a46 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0ca2f266188f4001abf68776b42f2f82'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:24.255933 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4562dc7e-6217-49bd-9e85-f227e2e59a46 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0ca2f266188f4001abf68776b42f2f82'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:24.268453 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1046/2105] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:27:23 2022] DELETE /identity/v3/users/0ca2f266188f4001abf68776b42f2f82 => generated 0 bytes in 301 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:27:24.270488 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d272864-fce8-4596-926b-a45bdb3f2a97 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:24.270635 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d272864-fce8-4596-926b-a45bdb3f2a97 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:24.270931 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d272864-fce8-4596-926b-a45bdb3f2a97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:24.299359 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-2d272864-fce8-4596-926b-a45bdb3f2a97 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:27:24.301263 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-2d272864-fce8-4596-926b-a45bdb3f2a97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:27:24.315549 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:27:24.422137 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1060/2106] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:27:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:27:24.427847 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4562dc7e-6217-49bd-9e85-f227e2e59a46 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:24.431293 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-4562dc7e-6217-49bd-9e85-f227e2e59a46 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:27:24.467716 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d272864-fce8-4596-926b-a45bdb3f2a97 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:24.469257 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-2d272864-fce8-4596-926b-a45bdb3f2a97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:27:24.524298 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4562dc7e-6217-49bd-9e85-f227e2e59a46 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:24.534553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366d6ad5-c466-4c60-b12e-dfa7eefbea4f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:24.535386 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366d6ad5-c466-4c60-b12e-dfa7eefbea4f service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:24.535768 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366d6ad5-c466-4c60-b12e-dfa7eefbea4f service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:24.536158 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366d6ad5-c466-4c60-b12e-dfa7eefbea4f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:24.552858 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-366d6ad5-c466-4c60-b12e-dfa7eefbea4f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:24.554758 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-366d6ad5-c466-4c60-b12e-dfa7eefbea4f service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:24.561796 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d272864-fce8-4596-926b-a45bdb3f2a97 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:24.568590 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1047/2107] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:27:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:27:24.573444 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e02ff718-75b4-45a3-8b16-bae613b0c26f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:24.574087 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02ff718-75b4-45a3-8b16-bae613b0c26f service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:24.574367 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02ff718-75b4-45a3-8b16-bae613b0c26f service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:24.574637 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02ff718-75b4-45a3-8b16-bae613b0c26f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:24.576571 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-e02ff718-75b4-45a3-8b16-bae613b0c26f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:27:24.707412 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e02ff718-75b4-45a3-8b16-bae613b0c26f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:24.708829 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e02ff718-75b4-45a3-8b16-bae613b0c26f service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:24.728183 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1061/2108] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:27:24 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 261 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 27 01:27:25.010943 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366d6ad5-c466-4c60-b12e-dfa7eefbea4f service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:25.012569 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-366d6ad5-c466-4c60-b12e-dfa7eefbea4f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:27:25.029210 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366d6ad5-c466-4c60-b12e-dfa7eefbea4f service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:25.035437 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f132686-a055-4664-ade4-2f8ab8caa5f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:25.036164 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f132686-a055-4664-ade4-2f8ab8caa5f0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:25.036572 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f132686-a055-4664-ade4-2f8ab8caa5f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:25.036843 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f132686-a055-4664-ade4-2f8ab8caa5f0 admin admin] PATH_INFO: `/v3/projects/6173634a570446f48fe4bf89887626df` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:25.046756 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f132686-a055-4664-ade4-2f8ab8caa5f0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6173634a570446f48fe4bf89887626df)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:25.048973 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f132686-a055-4664-ade4-2f8ab8caa5f0 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:25.102303 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:27:25.106649 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4f132686-a055-4664-ade4-2f8ab8caa5f0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6173634a570446f48fe4bf89887626df', 'request_id': 'req-4f132686-a055-4664-ade4-2f8ab8caa5f0'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:25.117816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4f132686-a055-4664-ade4-2f8ab8caa5f0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6173634a570446f48fe4bf89887626df', 'request_id': 'req-4f132686-a055-4664-ade4-2f8ab8caa5f0'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:25.126551 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4f132686-a055-4664-ade4-2f8ab8caa5f0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6173634a570446f48fe4bf89887626df', 'request_id': 'req-4f132686-a055-4664-ade4-2f8ab8caa5f0'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:25.139982 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4f132686-a055-4664-ade4-2f8ab8caa5f0 admin admin] The token cache is being invalidate because project 6173634a570446f48fe4bf89887626df was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:27:25.140278 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4f132686-a055-4664-ade4-2f8ab8caa5f0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:25.140938 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4f132686-a055-4664-ade4-2f8ab8caa5f0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:25.142499 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1048/2109] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:27:25 2022] DELETE /identity/v3/projects/6173634a570446f48fe4bf89887626df => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:27:28.796572 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e02ff718-75b4-45a3-8b16-bae613b0c26f service cinder] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:28.798128 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-e02ff718-75b4-45a3-8b16-bae613b0c26f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:27:28.814127 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e02ff718-75b4-45a3-8b16-bae613b0c26f service cinder] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:28.890241 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0034857-9418-4033-8f6b-27206fa0f755 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:28.890948 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0034857-9418-4033-8f6b-27206fa0f755 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:28.891235 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0034857-9418-4033-8f6b-27206fa0f755 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:28.891506 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0034857-9418-4033-8f6b-27206fa0f755 admin admin] PATH_INFO: `/v3/users/9c4406fa56f14b6b847872ed150c3b2b` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:28.892648 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0034857-9418-4033-8f6b-27206fa0f755 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9c4406fa56f14b6b847872ed150c3b2b)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:28.895051 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0034857-9418-4033-8f6b-27206fa0f755 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:28.906992 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:27:28.958061 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-f0034857-9418-4033-8f6b-27206fa0f755 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9c4406fa56f14b6b847872ed150c3b2b'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:28.969631 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-f0034857-9418-4033-8f6b-27206fa0f755 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9c4406fa56f14b6b847872ed150c3b2b'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:28.985725 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-f0034857-9418-4033-8f6b-27206fa0f755 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9c4406fa56f14b6b847872ed150c3b2b'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:28.986580 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-f0034857-9418-4033-8f6b-27206fa0f755 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9c4406fa56f14b6b847872ed150c3b2b'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:28.987343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-f0034857-9418-4033-8f6b-27206fa0f755 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9c4406fa56f14b6b847872ed150c3b2b'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:29.002788 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1062/2110] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:27:28 2022] DELETE /identity/v3/users/9c4406fa56f14b6b847872ed150c3b2b => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:27:29.007539 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30dbdb05-705c-49ea-a32c-679b87f9b6c0 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:29.008404 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30dbdb05-705c-49ea-a32c-679b87f9b6c0 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:29.008825 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30dbdb05-705c-49ea-a32c-679b87f9b6c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:29.029107 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-30dbdb05-705c-49ea-a32c-679b87f9b6c0 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:27:29.030674 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-30dbdb05-705c-49ea-a32c-679b87f9b6c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:27:29.108813 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1049/2111] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:27:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:27:29.116336 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0034857-9418-4033-8f6b-27206fa0f755 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:29.117802 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-f0034857-9418-4033-8f6b-27206fa0f755 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:27:29.186844 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0034857-9418-4033-8f6b-27206fa0f755 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:29.192526 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9498f050-9b47-48ca-8ed7-c8fbd30e79b0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:29.193295 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9498f050-9b47-48ca-8ed7-c8fbd30e79b0 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:29.193573 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9498f050-9b47-48ca-8ed7-c8fbd30e79b0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:29.193839 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9498f050-9b47-48ca-8ed7-c8fbd30e79b0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:29.205723 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9498f050-9b47-48ca-8ed7-c8fbd30e79b0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:29.208113 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9498f050-9b47-48ca-8ed7-c8fbd30e79b0 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:29.218647 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1063/2112] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:27:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 27 01:27:29.518380 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30dbdb05-705c-49ea-a32c-679b87f9b6c0 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:29.520613 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-30dbdb05-705c-49ea-a32c-679b87f9b6c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:27:29.533793 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30dbdb05-705c-49ea-a32c-679b87f9b6c0 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:29.539320 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f71e5998-d6af-4b89-a284-e734a3b39f59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:29.545106 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71e5998-d6af-4b89-a284-e734a3b39f59 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:29.545106 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71e5998-d6af-4b89-a284-e734a3b39f59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:29.545106 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71e5998-d6af-4b89-a284-e734a3b39f59 admin admin] PATH_INFO: `/v3/projects/d72ae16023c941a0ab5bc372859b81eb` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:29.545106 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f71e5998-d6af-4b89-a284-e734a3b39f59 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d72ae16023c941a0ab5bc372859b81eb)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:29.545106 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f71e5998-d6af-4b89-a284-e734a3b39f59 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:29.606564 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f71e5998-d6af-4b89-a284-e734a3b39f59 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd72ae16023c941a0ab5bc372859b81eb', 'request_id': 'req-f71e5998-d6af-4b89-a284-e734a3b39f59'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:29.623194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f71e5998-d6af-4b89-a284-e734a3b39f59 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd72ae16023c941a0ab5bc372859b81eb', 'request_id': 'req-f71e5998-d6af-4b89-a284-e734a3b39f59'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:29.633703 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f71e5998-d6af-4b89-a284-e734a3b39f59 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd72ae16023c941a0ab5bc372859b81eb', 'request_id': 'req-f71e5998-d6af-4b89-a284-e734a3b39f59'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:29.655928 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f71e5998-d6af-4b89-a284-e734a3b39f59 admin admin] The token cache is being invalidate because project d72ae16023c941a0ab5bc372859b81eb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:27:29.656874 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f71e5998-d6af-4b89-a284-e734a3b39f59 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:29.657575 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f71e5998-d6af-4b89-a284-e734a3b39f59 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:29.659131 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1050/2113] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:27:29 2022] DELETE /identity/v3/projects/d72ae16023c941a0ab5bc372859b81eb => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:27:29.664199 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9498f050-9b47-48ca-8ed7-c8fbd30e79b0 service neutron] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:29.691485 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9498f050-9b47-48ca-8ed7-c8fbd30e79b0 service neutron] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:29.760144 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e659a5d-773f-4696-adb9-1830c3fdafc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:29.760840 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e659a5d-773f-4696-adb9-1830c3fdafc7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:29.761094 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e659a5d-773f-4696-adb9-1830c3fdafc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:29.761361 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e659a5d-773f-4696-adb9-1830c3fdafc7 admin admin] PATH_INFO: `/v3/users/4b15ee275b134dcebb43ea15cb790214` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:29.762804 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e659a5d-773f-4696-adb9-1830c3fdafc7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4b15ee275b134dcebb43ea15cb790214)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:29.764735 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e659a5d-773f-4696-adb9-1830c3fdafc7 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:29.861120 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:DELETE=15,keystone:SELECT=84,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:27:29.876934 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-0e659a5d-773f-4696-adb9-1830c3fdafc7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4b15ee275b134dcebb43ea15cb790214'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:29.887147 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-0e659a5d-773f-4696-adb9-1830c3fdafc7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4b15ee275b134dcebb43ea15cb790214'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:29.905043 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-0e659a5d-773f-4696-adb9-1830c3fdafc7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4b15ee275b134dcebb43ea15cb790214'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:29.906898 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-0e659a5d-773f-4696-adb9-1830c3fdafc7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4b15ee275b134dcebb43ea15cb790214'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:29.907532 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-0e659a5d-773f-4696-adb9-1830c3fdafc7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4b15ee275b134dcebb43ea15cb790214'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:29.941595 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1064/2114] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:27:29 2022] DELETE /identity/v3/users/4b15ee275b134dcebb43ea15cb790214 => generated 0 bytes in 278 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:27:30.269952 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f71e5998-d6af-4b89-a284-e734a3b39f59 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:30.271530 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f71e5998-d6af-4b89-a284-e734a3b39f59 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:27:30.289975 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f71e5998-d6af-4b89-a284-e734a3b39f59 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:30.351214 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:DELETE=11,keystone:SELECT=87,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:27:30.359682 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5924a612-7118-4814-9c73-24c682017044 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:30.360671 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5924a612-7118-4814-9c73-24c682017044 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:30.361760 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5924a612-7118-4814-9c73-24c682017044 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:30.362048 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5924a612-7118-4814-9c73-24c682017044 admin admin] PATH_INFO: `/v3/projects/9667cf28520149a5b0fcaae38c380a7a` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:30.364331 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5924a612-7118-4814-9c73-24c682017044 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9667cf28520149a5b0fcaae38c380a7a)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:30.366288 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5924a612-7118-4814-9c73-24c682017044 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:30.420492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-5924a612-7118-4814-9c73-24c682017044 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9667cf28520149a5b0fcaae38c380a7a', 'request_id': 'req-5924a612-7118-4814-9c73-24c682017044'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:30.431155 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-5924a612-7118-4814-9c73-24c682017044 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9667cf28520149a5b0fcaae38c380a7a', 'request_id': 'req-5924a612-7118-4814-9c73-24c682017044'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:30.440884 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-5924a612-7118-4814-9c73-24c682017044 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9667cf28520149a5b0fcaae38c380a7a', 'request_id': 'req-5924a612-7118-4814-9c73-24c682017044'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:30.462777 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-5924a612-7118-4814-9c73-24c682017044 admin admin] The token cache is being invalidate because project 9667cf28520149a5b0fcaae38c380a7a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:27:30.463109 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-5924a612-7118-4814-9c73-24c682017044 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:30.463804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-5924a612-7118-4814-9c73-24c682017044 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:30.465649 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1051/2115] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:27:30 2022] DELETE /identity/v3/projects/9667cf28520149a5b0fcaae38c380a7a => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:27:30.469683 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e659a5d-773f-4696-adb9-1830c3fdafc7 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:30.471081 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-0e659a5d-773f-4696-adb9-1830c3fdafc7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:27:30.487066 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e659a5d-773f-4696-adb9-1830c3fdafc7 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:30.554015 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5859555-6596-46f5-bb99-46e1ed11a5ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:30.555301 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5859555-6596-46f5-bb99-46e1ed11a5ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:30.555570 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5859555-6596-46f5-bb99-46e1ed11a5ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:30.555839 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5859555-6596-46f5-bb99-46e1ed11a5ab admin admin] PATH_INFO: `/v3/users/459069db230b4072898de596db774996` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:30.557206 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5859555-6596-46f5-bb99-46e1ed11a5ab admin admin] RBAC: Authorizing `identity:delete_user(user_id=459069db230b4072898de596db774996)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:30.559323 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5859555-6596-46f5-bb99-46e1ed11a5ab admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:30.620722 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-b5859555-6596-46f5-bb99-46e1ed11a5ab admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '459069db230b4072898de596db774996'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:30.629509 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-b5859555-6596-46f5-bb99-46e1ed11a5ab admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '459069db230b4072898de596db774996'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:30.643050 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-b5859555-6596-46f5-bb99-46e1ed11a5ab admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '459069db230b4072898de596db774996'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:30.644132 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-b5859555-6596-46f5-bb99-46e1ed11a5ab admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '459069db230b4072898de596db774996'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:30.645230 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-b5859555-6596-46f5-bb99-46e1ed11a5ab admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '459069db230b4072898de596db774996'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:30.658242 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1065/2116] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:27:30 2022] DELETE /identity/v3/users/459069db230b4072898de596db774996 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:27:30.976105 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5924a612-7118-4814-9c73-24c682017044 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:30.977714 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-5924a612-7118-4814-9c73-24c682017044 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:27:30.996006 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5924a612-7118-4814-9c73-24c682017044 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:31.068794 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e203b06a-9e6d-4463-b1c0-fd7c07eeb8d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:31.069679 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e203b06a-9e6d-4463-b1c0-fd7c07eeb8d9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:31.070059 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e203b06a-9e6d-4463-b1c0-fd7c07eeb8d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:31.070428 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e203b06a-9e6d-4463-b1c0-fd7c07eeb8d9 admin admin] PATH_INFO: `/v3/projects/2e2b5ad2ee7c42f1823b3b99707c9d89` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:31.071749 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e203b06a-9e6d-4463-b1c0-fd7c07eeb8d9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2e2b5ad2ee7c42f1823b3b99707c9d89)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:31.073859 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e203b06a-9e6d-4463-b1c0-fd7c07eeb8d9 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:31.131988 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-e203b06a-9e6d-4463-b1c0-fd7c07eeb8d9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2e2b5ad2ee7c42f1823b3b99707c9d89', 'request_id': 'req-e203b06a-9e6d-4463-b1c0-fd7c07eeb8d9'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:31.142148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-e203b06a-9e6d-4463-b1c0-fd7c07eeb8d9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2e2b5ad2ee7c42f1823b3b99707c9d89', 'request_id': 'req-e203b06a-9e6d-4463-b1c0-fd7c07eeb8d9'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:31.149775 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-e203b06a-9e6d-4463-b1c0-fd7c07eeb8d9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2e2b5ad2ee7c42f1823b3b99707c9d89', 'request_id': 'req-e203b06a-9e6d-4463-b1c0-fd7c07eeb8d9'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:31.161915 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-e203b06a-9e6d-4463-b1c0-fd7c07eeb8d9 admin admin] The token cache is being invalidate because project 2e2b5ad2ee7c42f1823b3b99707c9d89 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:27:31.162421 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-e203b06a-9e6d-4463-b1c0-fd7c07eeb8d9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:31.163089 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-e203b06a-9e6d-4463-b1c0-fd7c07eeb8d9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:31.164707 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1052/2117] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:27:30 2022] DELETE /identity/v3/projects/2e2b5ad2ee7c42f1823b3b99707c9d89 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:27:31.169724 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63f3629-2755-4593-8765-c332fc104313 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:31.170287 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63f3629-2755-4593-8765-c332fc104313 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:31.177208 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63f3629-2755-4593-8765-c332fc104313 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:31.206440 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-a63f3629-2755-4593-8765-c332fc104313 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:27:31.208422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-a63f3629-2755-4593-8765-c332fc104313 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:27:31.235037 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:27:31.278598 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1066/2118] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:27:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:27:31.287122 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94076fa6-08ae-432f-a98c-b06154efe691 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:31.287643 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94076fa6-08ae-432f-a98c-b06154efe691 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:31.287643 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94076fa6-08ae-432f-a98c-b06154efe691 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:31.312265 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-94076fa6-08ae-432f-a98c-b06154efe691 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:27:31.314171 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-94076fa6-08ae-432f-a98c-b06154efe691 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:27:31.322871 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1053/2119] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:27:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:27:31.327146 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63f3629-2755-4593-8765-c332fc104313 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:31.337741 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63f3629-2755-4593-8765-c332fc104313 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:31.345085 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31fb5872-e962-404a-9c7e-62883f342812 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:31.345716 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fb5872-e962-404a-9c7e-62883f342812 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:31.345993 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fb5872-e962-404a-9c7e-62883f342812 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:31.346259 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fb5872-e962-404a-9c7e-62883f342812 admin admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:31.347548 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31fb5872-e962-404a-9c7e-62883f342812 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:27:31.348016 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31fb5872-e962-404a-9c7e-62883f342812 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:31.349858 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31fb5872-e962-404a-9c7e-62883f342812 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:31.367266 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1067/2120] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:27:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:27:31.368697 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94076fa6-08ae-432f-a98c-b06154efe691 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:31.378420 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94076fa6-08ae-432f-a98c-b06154efe691 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:31.384361 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01f740c-754d-423a-8d35-9330f768ca4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:31.384906 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01f740c-754d-423a-8d35-9330f768ca4c admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:31.385215 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01f740c-754d-423a-8d35-9330f768ca4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:31.386130 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01f740c-754d-423a-8d35-9330f768ca4c admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:31.386782 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a01f740c-754d-423a-8d35-9330f768ca4c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:31.388824 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a01f740c-754d-423a-8d35-9330f768ca4c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:31.399348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:27:31.422622 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1054/2121] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:27:31 2022] POST /identity/v3/projects => generated 371 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:27:31.426937 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31fb5872-e962-404a-9c7e-62883f342812 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:31.437594 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31fb5872-e962-404a-9c7e-62883f342812 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:31.502735 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a1b3d0-d91f-4461-a793-f06ba87008e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:31.503849 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a1b3d0-d91f-4461-a793-f06ba87008e7 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:31.505330 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a1b3d0-d91f-4461-a793-f06ba87008e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:31.505907 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a1b3d0-d91f-4461-a793-f06ba87008e7 admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:31.506785 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a1b3d0-d91f-4461-a793-f06ba87008e7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:31.508795 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a1b3d0-d91f-4461-a793-f06ba87008e7 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:31.536245 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1068/2122] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:27:31 2022] POST /identity/v3/users => generated 286 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:27:31.541390 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01f740c-754d-423a-8d35-9330f768ca4c admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:31.552196 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01f740c-754d-423a-8d35-9330f768ca4c admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:31.560565 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f32a733a-64fa-476e-a6bb-006a5fa3ebfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:31.564403 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32a733a-64fa-476e-a6bb-006a5fa3ebfb admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:31.564729 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32a733a-64fa-476e-a6bb-006a5fa3ebfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:31.565025 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32a733a-64fa-476e-a6bb-006a5fa3ebfb admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:31.566356 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f32a733a-64fa-476e-a6bb-006a5fa3ebfb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:31.567989 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f32a733a-64fa-476e-a6bb-006a5fa3ebfb admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:31.581105 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1055/2123] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:27:31 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:27:31.583641 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a1b3d0-d91f-4461-a793-f06ba87008e7 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:31.593825 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a1b3d0-d91f-4461-a793-f06ba87008e7 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:31.601840 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a69f48c-0f14-4ece-8bef-917ece2f5968 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:31.602591 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a69f48c-0f14-4ece-8bef-917ece2f5968 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:31.602942 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a69f48c-0f14-4ece-8bef-917ece2f5968 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:31.603422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a69f48c-0f14-4ece-8bef-917ece2f5968 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:31.604192 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a69f48c-0f14-4ece-8bef-917ece2f5968 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:31.606026 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a69f48c-0f14-4ece-8bef-917ece2f5968 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:31.618830 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1069/2124] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:27:31 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:27:31.623140 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f32a733a-64fa-476e-a6bb-006a5fa3ebfb admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:31.634863 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f32a733a-64fa-476e-a6bb-006a5fa3ebfb admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:31.643423 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02024957-794e-490d-860c-c00b63a6dcdd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:31.644211 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02024957-794e-490d-860c-c00b63a6dcdd admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:31.644630 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02024957-794e-490d-860c-c00b63a6dcdd admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:31.644974 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02024957-794e-490d-860c-c00b63a6dcdd admin admin] PATH_INFO: `/v3/projects/92fe08f777a44190b717ae4bf2f2a0ed/users/ff9d1a3fc43d4b9eb50c734589ab1a08/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:31.661361 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02024957-794e-490d-860c-c00b63a6dcdd admin admin] RBAC: Authorizing `identity:create_grant(project_id=92fe08f777a44190b717ae4bf2f2a0ed, user_id=ff9d1a3fc43d4b9eb50c734589ab1a08, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:31.664327 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02024957-794e-490d-860c-c00b63a6dcdd admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:31.680353 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1056/2125] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:27:31 2022] PUT /identity/v3/projects/92fe08f777a44190b717ae4bf2f2a0ed/users/ff9d1a3fc43d4b9eb50c734589ab1a08/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:27:31.691053 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48645593-c9fc-46c5-a9a3-7da3bce082e5 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:31.691718 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48645593-c9fc-46c5-a9a3-7da3bce082e5 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:31.692103 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48645593-c9fc-46c5-a9a3-7da3bce082e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:31.723721 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-48645593-c9fc-46c5-a9a3-7da3bce082e5 None None] MFA Rules not processed for user `ff9d1a3fc43d4b9eb50c734589ab1a08`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:27:31.725052 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-48645593-c9fc-46c5-a9a3-7da3bce082e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:27:31.809053 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1070/2126] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:27:31 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:27:31.823522 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02024957-794e-490d-860c-c00b63a6dcdd admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:31.825666 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-02024957-794e-490d-860c-c00b63a6dcdd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:27:31.888467 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02024957-794e-490d-860c-c00b63a6dcdd admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:31.892766 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be748b27-d374-47a5-b621-ea35a8922aa8 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2880dbb00eec471285d242e8183dd8d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:31.893518 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be748b27-d374-47a5-b621-ea35a8922aa8 service glance] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:31.894383 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be748b27-d374-47a5-b621-ea35a8922aa8 service glance] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:31.895001 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be748b27-d374-47a5-b621-ea35a8922aa8 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:31.905439 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be748b27-d374-47a5-b621-ea35a8922aa8 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:31.906928 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be748b27-d374-47a5-b621-ea35a8922aa8 service glance] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:31.918847 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1057/2127] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:27:31 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:27:35.016287 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb4e2e5-f8f4-4e95-94db-dfcdceac67e1 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:35.016843 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb4e2e5-f8f4-4e95-94db-dfcdceac67e1 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:35.016843 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb4e2e5-f8f4-4e95-94db-dfcdceac67e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:35.037696 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-ceb4e2e5-f8f4-4e95-94db-dfcdceac67e1 None None] MFA Rules not processed for user `7a3f8b5eb09f4ba29f4bd343bd91f6a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:27:35.038900 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-ceb4e2e5-f8f4-4e95-94db-dfcdceac67e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:27:35.058133 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:27:35.095872 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1071/2128] 10.222.0.42 () {58 vars in 967 bytes} [Thu Oct 27 01:27:35 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:27:35.108487 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be748b27-d374-47a5-b621-ea35a8922aa8 service glance] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:35.109902 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-be748b27-d374-47a5-b621-ea35a8922aa8 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:27:35.182556 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be748b27-d374-47a5-b621-ea35a8922aa8 service glance] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:35.187459 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e261db9-997c-413e-9ac5-b88564de3c1a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77f2cf9e0c3541f9bccfb7e213d88609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:35.188052 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e261db9-997c-413e-9ac5-b88564de3c1a service swift] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:35.188323 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e261db9-997c-413e-9ac5-b88564de3c1a service swift] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:35.188591 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e261db9-997c-413e-9ac5-b88564de3c1a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:35.196861 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e261db9-997c-413e-9ac5-b88564de3c1a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:35.198494 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e261db9-997c-413e-9ac5-b88564de3c1a service swift] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:35.208566 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1058/2129] 10.222.0.42 () {60 vars in 1384 bytes} [Thu Oct 27 01:27:35 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 27 01:27:37.137000 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb4e2e5-f8f4-4e95-94db-dfcdceac67e1 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:37.138560 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-ceb4e2e5-f8f4-4e95-94db-dfcdceac67e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:27:37.207842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb4e2e5-f8f4-4e95-94db-dfcdceac67e1 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:37.212237 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf077552-3457-4a1e-a404-267814da5b26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:37.212846 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf077552-3457-4a1e-a404-267814da5b26 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:37.213147 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf077552-3457-4a1e-a404-267814da5b26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:37.213417 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf077552-3457-4a1e-a404-267814da5b26 admin admin] PATH_INFO: `/v3/users/1505f37c15c0461a912d59172ebdf94a` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:37.214534 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf077552-3457-4a1e-a404-267814da5b26 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1505f37c15c0461a912d59172ebdf94a)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:37.216454 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf077552-3457-4a1e-a404-267814da5b26 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:37.265400 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-cf077552-3457-4a1e-a404-267814da5b26 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1505f37c15c0461a912d59172ebdf94a'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:37.273824 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-cf077552-3457-4a1e-a404-267814da5b26 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1505f37c15c0461a912d59172ebdf94a'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:37.285951 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-cf077552-3457-4a1e-a404-267814da5b26 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1505f37c15c0461a912d59172ebdf94a'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:37.286589 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-cf077552-3457-4a1e-a404-267814da5b26 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1505f37c15c0461a912d59172ebdf94a'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:37.287190 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-cf077552-3457-4a1e-a404-267814da5b26 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1505f37c15c0461a912d59172ebdf94a'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:37.299253 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1072/2130] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:27:37 2022] DELETE /identity/v3/users/1505f37c15c0461a912d59172ebdf94a => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:27:37.475816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e261db9-997c-413e-9ac5-b88564de3c1a service swift] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:37.486610 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e261db9-997c-413e-9ac5-b88564de3c1a service swift] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:37.535164 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:27:37.548583 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-688bcaaa-bbc3-4d61-9d66-52f8f8295bdb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:37.549294 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688bcaaa-bbc3-4d61-9d66-52f8f8295bdb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:37.549569 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688bcaaa-bbc3-4d61-9d66-52f8f8295bdb admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:37.549834 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688bcaaa-bbc3-4d61-9d66-52f8f8295bdb admin admin] PATH_INFO: `/v3/projects/a816bcfd87b34071a24b948c351811c4` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:37.551515 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-688bcaaa-bbc3-4d61-9d66-52f8f8295bdb admin admin] RBAC: Authorizing `identity:delete_project(project_id=a816bcfd87b34071a24b948c351811c4)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:37.553970 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-688bcaaa-bbc3-4d61-9d66-52f8f8295bdb admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:37.599515 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-688bcaaa-bbc3-4d61-9d66-52f8f8295bdb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a816bcfd87b34071a24b948c351811c4', 'request_id': 'req-688bcaaa-bbc3-4d61-9d66-52f8f8295bdb'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:37.607228 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-688bcaaa-bbc3-4d61-9d66-52f8f8295bdb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a816bcfd87b34071a24b948c351811c4', 'request_id': 'req-688bcaaa-bbc3-4d61-9d66-52f8f8295bdb'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:37.613765 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-688bcaaa-bbc3-4d61-9d66-52f8f8295bdb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a816bcfd87b34071a24b948c351811c4', 'request_id': 'req-688bcaaa-bbc3-4d61-9d66-52f8f8295bdb'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:37.624442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-688bcaaa-bbc3-4d61-9d66-52f8f8295bdb admin admin] The token cache is being invalidate because project a816bcfd87b34071a24b948c351811c4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:27:37.624765 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-688bcaaa-bbc3-4d61-9d66-52f8f8295bdb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:37.625562 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-688bcaaa-bbc3-4d61-9d66-52f8f8295bdb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:27:37.627271 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1059/2131] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:27:37 2022] DELETE /identity/v3/projects/a816bcfd87b34071a24b948c351811c4 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:27:47.295828 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=67,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:27:47.621304 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=30,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:27:51.276108 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8b019a-f6d0-4c17-a9f9-bf8f964f6ab7 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:51.276581 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8b019a-f6d0-4c17-a9f9-bf8f964f6ab7 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:51.276689 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8b019a-f6d0-4c17-a9f9-bf8f964f6ab7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:51.296128 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-4a8b019a-f6d0-4c17-a9f9-bf8f964f6ab7 None None] MFA Rules not processed for user `99bcd22ff73b4526a54cf0e76f509436`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:27:51.297335 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-4a8b019a-f6d0-4c17-a9f9-bf8f964f6ab7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:27:51.351332 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1073/2132] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:27:51 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:27:52.438228 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c87dd47-c329-4a7f-bf0e-024d7b403e71 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:52.438687 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c87dd47-c329-4a7f-bf0e-024d7b403e71 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:52.438803 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c87dd47-c329-4a7f-bf0e-024d7b403e71 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:52.440105 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1060/2133] 10.222.0.42 () {54 vars in 961 bytes} [Thu Oct 27 01:27:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:27:52.445953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5dc68e-c793-48b2-94b0-b4c71dfc1f0e None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:52.446362 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5dc68e-c793-48b2-94b0-b4c71dfc1f0e None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:52.446494 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5dc68e-c793-48b2-94b0-b4c71dfc1f0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:52.467505 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-ca5dc68e-c793-48b2-94b0-b4c71dfc1f0e None None] MFA Rules not processed for user `62051562fc0b4cd08750d2293e71787c`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:27:52.468679 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-ca5dc68e-c793-48b2-94b0-b4c71dfc1f0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:27:52.531076 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1074/2134] 10.222.0.42 () {58 vars in 1058 bytes} [Thu Oct 27 01:27:52 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:27:52.539079 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c87dd47-c329-4a7f-bf0e-024d7b403e71 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:27:52.552864 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c87dd47-c329-4a7f-bf0e-024d7b403e71 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:27:52.558327 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-471749af-2508-455c-9e49-1ca55954bc25 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62051562fc0b4cd08750d2293e71787c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:27:52.558907 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471749af-2508-455c-9e49-1ca55954bc25 service placement] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:27:52.559177 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471749af-2508-455c-9e49-1ca55954bc25 service placement] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:27:52.559449 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471749af-2508-455c-9e49-1ca55954bc25 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:27:52.561195 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-471749af-2508-455c-9e49-1ca55954bc25 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:27:52.572974 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-471749af-2508-455c-9e49-1ca55954bc25 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:27:52.574587 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-471749af-2508-455c-9e49-1ca55954bc25 service placement] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:27:52.584559 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1061/2135] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:27:52 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 27 01:28:02.522739 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=54 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:28:02.580234 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=6 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:28:12.737474 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5dc68e-c793-48b2-94b0-b4c71dfc1f0e None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:28:12.739040 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-ca5dc68e-c793-48b2-94b0-b4c71dfc1f0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:28:12.802722 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5dc68e-c793-48b2-94b0-b4c71dfc1f0e None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:28:12.807425 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4671c550-7cbd-4bb1-ba81-be80491203e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:28:12.808042 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4671c550-7cbd-4bb1-ba81-be80491203e6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:28:12.808314 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4671c550-7cbd-4bb1-ba81-be80491203e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:28:12.808579 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4671c550-7cbd-4bb1-ba81-be80491203e6 admin admin] PATH_INFO: `/v3/users/ff9d1a3fc43d4b9eb50c734589ab1a08` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:28:12.809708 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4671c550-7cbd-4bb1-ba81-be80491203e6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ff9d1a3fc43d4b9eb50c734589ab1a08)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:28:12.811651 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4671c550-7cbd-4bb1-ba81-be80491203e6 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:28:12.857968 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-4671c550-7cbd-4bb1-ba81-be80491203e6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ff9d1a3fc43d4b9eb50c734589ab1a08'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:28:12.865710 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-4671c550-7cbd-4bb1-ba81-be80491203e6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ff9d1a3fc43d4b9eb50c734589ab1a08'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:28:12.877636 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-4671c550-7cbd-4bb1-ba81-be80491203e6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ff9d1a3fc43d4b9eb50c734589ab1a08'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:28:12.878267 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-4671c550-7cbd-4bb1-ba81-be80491203e6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ff9d1a3fc43d4b9eb50c734589ab1a08'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:28:12.878993 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-4671c550-7cbd-4bb1-ba81-be80491203e6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ff9d1a3fc43d4b9eb50c734589ab1a08'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:28:12.891024 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1075/2136] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:28:12 2022] DELETE /identity/v3/users/ff9d1a3fc43d4b9eb50c734589ab1a08 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:28:12.896249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d15c914-2016-4c08-9b95-d192a11bf8b2 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:28:12.896717 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d15c914-2016-4c08-9b95-d192a11bf8b2 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:28:12.896813 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d15c914-2016-4c08-9b95-d192a11bf8b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:28:12.916633 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-8d15c914-2016-4c08-9b95-d192a11bf8b2 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:28:12.917877 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-8d15c914-2016-4c08-9b95-d192a11bf8b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:28:12.975958 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1062/2137] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:28:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:28:12.985658 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4671c550-7cbd-4bb1-ba81-be80491203e6 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:28:12.987059 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-4671c550-7cbd-4bb1-ba81-be80491203e6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:28:13.044087 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4671c550-7cbd-4bb1-ba81-be80491203e6 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:28:13.047682 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5c8d32-74b3-4ec5-8b0e-89c011c9b132 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:28:13.048276 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5c8d32-74b3-4ec5-8b0e-89c011c9b132 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:28:13.048552 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5c8d32-74b3-4ec5-8b0e-89c011c9b132 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:28:13.048821 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5c8d32-74b3-4ec5-8b0e-89c011c9b132 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:28:13.057901 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5c8d32-74b3-4ec5-8b0e-89c011c9b132 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:28:13.059304 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5c8d32-74b3-4ec5-8b0e-89c011c9b132 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:28:13.069194 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1076/2138] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:28:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:28:13.354436 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d15c914-2016-4c08-9b95-d192a11bf8b2 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:28:13.355840 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-8d15c914-2016-4c08-9b95-d192a11bf8b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:28:13.367874 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d15c914-2016-4c08-9b95-d192a11bf8b2 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:28:13.372537 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b1ee6e-715e-41b4-be8c-178baa14847e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:28:13.373123 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b1ee6e-715e-41b4-be8c-178baa14847e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:28:13.373420 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b1ee6e-715e-41b4-be8c-178baa14847e admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:28:13.373672 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b1ee6e-715e-41b4-be8c-178baa14847e admin admin] PATH_INFO: `/v3/projects/92fe08f777a44190b717ae4bf2f2a0ed` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:28:13.374780 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b1ee6e-715e-41b4-be8c-178baa14847e admin admin] RBAC: Authorizing `identity:delete_project(project_id=92fe08f777a44190b717ae4bf2f2a0ed)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:28:13.376681 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b1ee6e-715e-41b4-be8c-178baa14847e admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:28:13.424182 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-a8b1ee6e-715e-41b4-be8c-178baa14847e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '92fe08f777a44190b717ae4bf2f2a0ed', 'request_id': 'req-a8b1ee6e-715e-41b4-be8c-178baa14847e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:28:13.431594 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-a8b1ee6e-715e-41b4-be8c-178baa14847e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '92fe08f777a44190b717ae4bf2f2a0ed', 'request_id': 'req-a8b1ee6e-715e-41b4-be8c-178baa14847e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:28:13.437451 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-a8b1ee6e-715e-41b4-be8c-178baa14847e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '92fe08f777a44190b717ae4bf2f2a0ed', 'request_id': 'req-a8b1ee6e-715e-41b4-be8c-178baa14847e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:28:13.448814 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-a8b1ee6e-715e-41b4-be8c-178baa14847e admin admin] The token cache is being invalidate because project 92fe08f777a44190b717ae4bf2f2a0ed was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:28:13.449136 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-a8b1ee6e-715e-41b4-be8c-178baa14847e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:28:13.449761 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-a8b1ee6e-715e-41b4-be8c-178baa14847e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:28:13.451233 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1063/2139] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:28:13 2022] DELETE /identity/v3/projects/92fe08f777a44190b717ae4bf2f2a0ed => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:28:23.064462 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=79,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:28:23.445895 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=54,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:29:01.191543 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5c8d32-74b3-4ec5-8b0e-89c011c9b132 service neutron] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:29:01.193705 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-8a5c8d32-74b3-4ec5-8b0e-89c011c9b132 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:29:01.251260 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5c8d32-74b3-4ec5-8b0e-89c011c9b132 service neutron] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:29:01.299801 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b31fb226-a24b-436c-89f1-6a66abbec0ad service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:29:01.300397 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31fb226-a24b-436c-89f1-6a66abbec0ad service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:29:01.300672 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31fb226-a24b-436c-89f1-6a66abbec0ad service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:29:01.300939 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31fb226-a24b-436c-89f1-6a66abbec0ad service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:29:01.303095 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-b31fb226-a24b-436c-89f1-6a66abbec0ad service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:29:01.372227 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b31fb226-a24b-436c-89f1-6a66abbec0ad service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:29:01.373710 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b31fb226-a24b-436c-89f1-6a66abbec0ad service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:29:01.385258 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1077/2140] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:29:01 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 194 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:29:01.543844 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b1ee6e-715e-41b4-be8c-178baa14847e admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:29:01.545444 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-a8b1ee6e-715e-41b4-be8c-178baa14847e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:29:01.604375 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b1ee6e-715e-41b4-be8c-178baa14847e admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:29:01.609103 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d173c38-4486-45f2-8ccc-be013a00a9e8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:29:01.609382 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d173c38-4486-45f2-8ccc-be013a00a9e8 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:29:01.609924 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d173c38-4486-45f2-8ccc-be013a00a9e8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:29:01.610199 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d173c38-4486-45f2-8ccc-be013a00a9e8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:29:01.611902 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-2d173c38-4486-45f2-8ccc-be013a00a9e8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:29:01.625894 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d173c38-4486-45f2-8ccc-be013a00a9e8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:29:01.627320 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d173c38-4486-45f2-8ccc-be013a00a9e8 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:29:01.638850 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1064/2141] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:29:01 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:29:07.464916 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b31fb226-a24b-436c-89f1-6a66abbec0ad service cinder] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:29:07.466496 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-b31fb226-a24b-436c-89f1-6a66abbec0ad service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:29:07.480563 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b31fb226-a24b-436c-89f1-6a66abbec0ad service cinder] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:29:07.485864 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20907cf9-780a-4508-bd6b-0697de989f45 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:29:07.486446 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20907cf9-780a-4508-bd6b-0697de989f45 service nova] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:29:07.486719 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20907cf9-780a-4508-bd6b-0697de989f45 service nova] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:29:07.486984 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20907cf9-780a-4508-bd6b-0697de989f45 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:29:07.488598 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-20907cf9-780a-4508-bd6b-0697de989f45 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:29:07.543305 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:29:07.545834 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20907cf9-780a-4508-bd6b-0697de989f45 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:29:07.547924 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20907cf9-780a-4508-bd6b-0697de989f45 service nova] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:29:07.558039 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1078/2142] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:29:07 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:29:11.633965 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=26 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:29:17.553805 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=2 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:29:19.345501 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d173c38-4486-45f2-8ccc-be013a00a9e8 service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:29:19.347114 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-2d173c38-4486-45f2-8ccc-be013a00a9e8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:29:19.357782 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d173c38-4486-45f2-8ccc-be013a00a9e8 service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:29:19.361443 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3469f449-863e-440e-844e-0db8fc1784aa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:29:19.362038 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3469f449-863e-440e-844e-0db8fc1784aa service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:29:19.362316 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3469f449-863e-440e-844e-0db8fc1784aa service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:29:19.362585 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3469f449-863e-440e-844e-0db8fc1784aa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:29:19.364420 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-3469f449-863e-440e-844e-0db8fc1784aa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:29:19.374243 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3469f449-863e-440e-844e-0db8fc1784aa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:29:19.375671 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3469f449-863e-440e-844e-0db8fc1784aa service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:29:19.384380 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1065/2143] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:29:19 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:29:21.807359 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a116e1d7-08e5-416b-9507-a8d414a4d3d4 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:29:21.807876 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a116e1d7-08e5-416b-9507-a8d414a4d3d4 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:29:21.807876 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a116e1d7-08e5-416b-9507-a8d414a4d3d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:29:21.826778 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-a116e1d7-08e5-416b-9507-a8d414a4d3d4 None None] MFA Rules not processed for user `d311fc1f82124819b509f766fa432e2c`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:29:21.827961 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-a116e1d7-08e5-416b-9507-a8d414a4d3d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:29:21.835264 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1079/2144] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:29:21 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:29:25.994698 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3469f449-863e-440e-844e-0db8fc1784aa service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:29:26.010528 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3469f449-863e-440e-844e-0db8fc1784aa service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:29:26.016055 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda31720-0b11-4db2-820d-595aeb9de57c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:29:26.016839 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda31720-0b11-4db2-820d-595aeb9de57c service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:29:26.017152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda31720-0b11-4db2-820d-595aeb9de57c service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:29:26.017476 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda31720-0b11-4db2-820d-595aeb9de57c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:29:26.019276 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-fda31720-0b11-4db2-820d-595aeb9de57c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:29:26.081024 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda31720-0b11-4db2-820d-595aeb9de57c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:29:26.082636 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda31720-0b11-4db2-820d-595aeb9de57c service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:29:26.092435 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1066/2145] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:29:25 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:29:26.998362 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a116e1d7-08e5-416b-9507-a8d414a4d3d4 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:29:27.000640 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-a116e1d7-08e5-416b-9507-a8d414a4d3d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:29:27.023759 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a116e1d7-08e5-416b-9507-a8d414a4d3d4 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:29:27.029249 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d285c127-d63e-45cf-ba71-aba7109e40bc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:29:27.029851 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d285c127-d63e-45cf-ba71-aba7109e40bc service nova] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:29:27.030125 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d285c127-d63e-45cf-ba71-aba7109e40bc service nova] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:29:27.030389 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d285c127-d63e-45cf-ba71-aba7109e40bc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:29:27.032249 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-d285c127-d63e-45cf-ba71-aba7109e40bc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:29:27.044409 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d285c127-d63e-45cf-ba71-aba7109e40bc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:29:27.046008 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d285c127-d63e-45cf-ba71-aba7109e40bc service nova] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:29:27.058630 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1080/2146] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:29:26 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 61 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:29:36.088077 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=34 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:29:37.052883 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=9 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:29:39.017912 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda31720-0b11-4db2-820d-595aeb9de57c service nova] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:29:39.029433 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda31720-0b11-4db2-820d-595aeb9de57c service nova] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:29:39.035071 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-532fc541-996f-40f8-8c13-2953bdabb967 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:29:39.035696 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532fc541-996f-40f8-8c13-2953bdabb967 service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:29:39.036000 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532fc541-996f-40f8-8c13-2953bdabb967 service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:29:39.036275 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532fc541-996f-40f8-8c13-2953bdabb967 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:29:39.038094 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-532fc541-996f-40f8-8c13-2953bdabb967 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:29:39.100959 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-532fc541-996f-40f8-8c13-2953bdabb967 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:29:39.102614 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-532fc541-996f-40f8-8c13-2953bdabb967 service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:29:39.112669 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1067/2147] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:29:39 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:29:42.663534 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d285c127-d63e-45cf-ba71-aba7109e40bc service nova] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:29:42.675756 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d285c127-d63e-45cf-ba71-aba7109e40bc service nova] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:29:42.682429 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6ddf1c-a048-44b4-9694-a8794cc85a79 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:29:42.683382 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6ddf1c-a048-44b4-9694-a8794cc85a79 service nova] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:29:42.683904 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6ddf1c-a048-44b4-9694-a8794cc85a79 service nova] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:29:42.684379 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6ddf1c-a048-44b4-9694-a8794cc85a79 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:29:42.686869 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-6f6ddf1c-a048-44b4-9694-a8794cc85a79 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:29:42.700842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f6ddf1c-a048-44b4-9694-a8794cc85a79 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:29:42.702929 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f6ddf1c-a048-44b4-9694-a8794cc85a79 service nova] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:29:42.715915 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1081/2148] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:29:42 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:29:49.107854 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=28 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:29:52.711382 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=6 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:29:52.803360 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73aaaa9d-7d9d-4820-832b-50b5808901e4 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:29:52.803883 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73aaaa9d-7d9d-4820-832b-50b5808901e4 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:29:52.803883 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73aaaa9d-7d9d-4820-832b-50b5808901e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:29:52.825617 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-73aaaa9d-7d9d-4820-832b-50b5808901e4 None None] MFA Rules not processed for user `99bcd22ff73b4526a54cf0e76f509436`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:29:52.826837 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-73aaaa9d-7d9d-4820-832b-50b5808901e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:29:52.833890 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1068/2149] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:29:52 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:29:53.133465 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6ddf1c-a048-44b4-9694-a8794cc85a79 service nova] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:29:53.142428 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6ddf1c-a048-44b4-9694-a8794cc85a79 service nova] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:29:53.146160 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2bab40-59f8-44d0-bb50-71b11d451bd2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:29:53.146722 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2bab40-59f8-44d0-bb50-71b11d451bd2 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:29:53.146993 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2bab40-59f8-44d0-bb50-71b11d451bd2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:29:53.147258 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2bab40-59f8-44d0-bb50-71b11d451bd2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:29:53.148960 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-af2bab40-59f8-44d0-bb50-71b11d451bd2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:29:53.158320 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af2bab40-59f8-44d0-bb50-71b11d451bd2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:29:53.159713 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af2bab40-59f8-44d0-bb50-71b11d451bd2 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:29:53.167411 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1082/2150] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:29:53 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:29:53.482990 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73aaaa9d-7d9d-4820-832b-50b5808901e4 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:29:53.484468 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-73aaaa9d-7d9d-4820-832b-50b5808901e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:29:53.557786 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73aaaa9d-7d9d-4820-832b-50b5808901e4 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:29:53.562727 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce055183-3472-417f-94ed-6e0560fc6e4c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62051562fc0b4cd08750d2293e71787c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:29:53.563324 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce055183-3472-417f-94ed-6e0560fc6e4c service placement] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:29:53.563601 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce055183-3472-417f-94ed-6e0560fc6e4c service placement] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:29:53.563869 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce055183-3472-417f-94ed-6e0560fc6e4c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:29:53.565592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ce055183-3472-417f-94ed-6e0560fc6e4c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:29:53.577305 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce055183-3472-417f-94ed-6e0560fc6e4c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:29:53.578875 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce055183-3472-417f-94ed-6e0560fc6e4c service placement] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:29:53.588720 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1069/2151] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:29:53 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:30:01.245937 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2bab40-59f8-44d0-bb50-71b11d451bd2 service neutron] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:30:01.259706 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2bab40-59f8-44d0-bb50-71b11d451bd2 service neutron] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:30:01.265267 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f316156c-6d6c-46e9-9453-fe658886266b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:30:01.265863 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f316156c-6d6c-46e9-9453-fe658886266b service nova] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:30:01.266141 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f316156c-6d6c-46e9-9453-fe658886266b service nova] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:30:01.266403 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f316156c-6d6c-46e9-9453-fe658886266b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:30:01.268103 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-f316156c-6d6c-46e9-9453-fe658886266b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:30:01.281166 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f316156c-6d6c-46e9-9453-fe658886266b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:30:01.282748 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f316156c-6d6c-46e9-9453-fe658886266b service nova] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:30:01.296831 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1083/2152] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:30:01 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:30:03.584510 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=35 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:30:11.291958 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=12 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:30:33.018461 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a7397b-5fae-4e3f-b093-ab18266b8959 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:30:33.019257 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a7397b-5fae-4e3f-b093-ab18266b8959 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:30:33.019257 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a7397b-5fae-4e3f-b093-ab18266b8959 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:30:33.020410 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1070/2153] 10.222.0.42 () {54 vars in 921 bytes} [Thu Oct 27 01:30:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:30:33.026969 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd45a9f-8c89-496b-9534-9673d8ca6cd0 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:30:33.027463 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd45a9f-8c89-496b-9534-9673d8ca6cd0 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:30:33.027566 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd45a9f-8c89-496b-9534-9673d8ca6cd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:30:33.047967 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-cfd45a9f-8c89-496b-9534-9673d8ca6cd0 None None] MFA Rules not processed for user `4072417990c94b0488871332d28224ba`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:30:33.049233 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-cfd45a9f-8c89-496b-9534-9673d8ca6cd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:30:33.056370 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1084/2154] 10.222.0.42 () {58 vars in 1018 bytes} [Thu Oct 27 01:30:33 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:30:33.070707 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a7397b-5fae-4e3f-b093-ab18266b8959 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:30:33.084345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a7397b-5fae-4e3f-b093-ab18266b8959 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:30:33.088625 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a69cd17-63b4-46e2-8eba-b6bbb7fc338a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:30:33.089454 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a69cd17-63b4-46e2-8eba-b6bbb7fc338a service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:30:33.089867 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a69cd17-63b4-46e2-8eba-b6bbb7fc338a service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:30:33.090225 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a69cd17-63b4-46e2-8eba-b6bbb7fc338a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:30:33.098173 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a69cd17-63b4-46e2-8eba-b6bbb7fc338a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:30:33.099734 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a69cd17-63b4-46e2-8eba-b6bbb7fc338a service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:30:33.122683 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1071/2155] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:30:33 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:30:34.222320 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1a56ab-b768-42ca-b4c2-5fd4d4d260c7 None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:30:34.222825 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1a56ab-b768-42ca-b4c2-5fd4d4d260c7 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:30:34.222825 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1a56ab-b768-42ca-b4c2-5fd4d4d260c7 None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:30:34.224183 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1085/2156] 10.222.0.42 () {54 vars in 921 bytes} [Thu Oct 27 01:30:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:30:34.229172 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42857e2-ec02-4518-a716-bf2805d4e77c None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:30:34.229621 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42857e2-ec02-4518-a716-bf2805d4e77c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:30:34.229747 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42857e2-ec02-4518-a716-bf2805d4e77c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:30:34.249834 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-b42857e2-ec02-4518-a716-bf2805d4e77c None None] MFA Rules not processed for user `51033d56f85f4282a49ded6338440677`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:30:34.251064 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-b42857e2-ec02-4518-a716-bf2805d4e77c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:30:34.261476 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1072/2157] 10.222.0.42 () {58 vars in 1018 bytes} [Thu Oct 27 01:30:34 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:30:34.272612 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff1a56ab-b768-42ca-b4c2-5fd4d4d260c7 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:30:34.283593 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff1a56ab-b768-42ca-b4c2-5fd4d4d260c7 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:30:34.287464 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762dfb8b-f745-4e3d-9910-ac1f81d79df5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:30:34.288088 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762dfb8b-f745-4e3d-9910-ac1f81d79df5 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:30:34.288366 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762dfb8b-f745-4e3d-9910-ac1f81d79df5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:30:34.288636 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762dfb8b-f745-4e3d-9910-ac1f81d79df5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:30:34.299348 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-762dfb8b-f745-4e3d-9910-ac1f81d79df5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:30:34.301745 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-762dfb8b-f745-4e3d-9910-ac1f81d79df5 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:30:34.312936 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1086/2158] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:30:34 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:30:44.244698 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=9 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:30:44.308511 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=9 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:30:51.834462 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42857e2-ec02-4518-a716-bf2805d4e77c None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:30:51.843956 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42857e2-ec02-4518-a716-bf2805d4e77c None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:30:51.848775 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8dac809-302a-42cb-a0e8-2ee41a3567e7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:30:51.849490 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dac809-302a-42cb-a0e8-2ee41a3567e7 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:30:51.849903 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dac809-302a-42cb-a0e8-2ee41a3567e7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:30:51.850201 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dac809-302a-42cb-a0e8-2ee41a3567e7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:30:51.852202 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e8dac809-302a-42cb-a0e8-2ee41a3567e7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:30:51.935077 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8dac809-302a-42cb-a0e8-2ee41a3567e7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:30:51.936808 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8dac809-302a-42cb-a0e8-2ee41a3567e7 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:30:51.950074 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1073/2159] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:30:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:30:55.079171 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12681479-edc5-4c30-bd8e-71dca4e8e2e5 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:30:55.079690 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12681479-edc5-4c30-bd8e-71dca4e8e2e5 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:30:55.079840 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12681479-edc5-4c30-bd8e-71dca4e8e2e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:30:55.104238 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-12681479-edc5-4c30-bd8e-71dca4e8e2e5 None None] MFA Rules not processed for user `cca36a277e3643a498b27e3c5d5e14f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:30:55.106950 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-12681479-edc5-4c30-bd8e-71dca4e8e2e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:30:55.119720 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1087/2160] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:30:55 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:30:56.025364 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8dac809-302a-42cb-a0e8-2ee41a3567e7 service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:30:56.027373 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e8dac809-302a-42cb-a0e8-2ee41a3567e7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:30:56.046122 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8dac809-302a-42cb-a0e8-2ee41a3567e7 service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:30:56.056544 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679e9ba0-b2e8-456f-937a-88e69a766944 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62051562fc0b4cd08750d2293e71787c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:30:56.057366 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679e9ba0-b2e8-456f-937a-88e69a766944 service placement] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:30:56.057797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679e9ba0-b2e8-456f-937a-88e69a766944 service placement] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:30:56.058241 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679e9ba0-b2e8-456f-937a-88e69a766944 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:30:56.061220 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-679e9ba0-b2e8-456f-937a-88e69a766944 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:30:56.078241 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679e9ba0-b2e8-456f-937a-88e69a766944 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:30:56.080923 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679e9ba0-b2e8-456f-937a-88e69a766944 service placement] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:30:56.099163 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1074/2161] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:30:56 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 74 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:30:56.108636 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12681479-edc5-4c30-bd8e-71dca4e8e2e5 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:30:56.110276 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-12681479-edc5-4c30-bd8e-71dca4e8e2e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:30:56.132507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12681479-edc5-4c30-bd8e-71dca4e8e2e5 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:30:56.137199 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574630f2-f2f7-4c12-af9d-fb2b090d6e94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:30:56.138218 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574630f2-f2f7-4c12-af9d-fb2b090d6e94 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:30:56.138665 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574630f2-f2f7-4c12-af9d-fb2b090d6e94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:30:56.139270 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574630f2-f2f7-4c12-af9d-fb2b090d6e94 admin admin] PATH_INFO: `/v3/users/d311fc1f82124819b509f766fa432e2c` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:30:56.140892 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574630f2-f2f7-4c12-af9d-fb2b090d6e94 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d311fc1f82124819b509f766fa432e2c)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:30:56.144048 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574630f2-f2f7-4c12-af9d-fb2b090d6e94 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:30:56.203352 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-574630f2-f2f7-4c12-af9d-fb2b090d6e94 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd311fc1f82124819b509f766fa432e2c'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:30:56.214803 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-574630f2-f2f7-4c12-af9d-fb2b090d6e94 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd311fc1f82124819b509f766fa432e2c'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:30:56.233940 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-574630f2-f2f7-4c12-af9d-fb2b090d6e94 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd311fc1f82124819b509f766fa432e2c'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:30:56.234824 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-574630f2-f2f7-4c12-af9d-fb2b090d6e94 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd311fc1f82124819b509f766fa432e2c'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:30:56.235479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-574630f2-f2f7-4c12-af9d-fb2b090d6e94 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd311fc1f82124819b509f766fa432e2c'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:30:56.249748 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1088/2162] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:30:56 2022] DELETE /identity/v3/users/d311fc1f82124819b509f766fa432e2c => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:30:56.453946 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679e9ba0-b2e8-456f-937a-88e69a766944 service placement] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:30:56.466894 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679e9ba0-b2e8-456f-937a-88e69a766944 service placement] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:30:56.533764 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f8ea57-ebcd-4755-b1f1-5fbe91ef7940 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:30:56.535217 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f8ea57-ebcd-4755-b1f1-5fbe91ef7940 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:30:56.535521 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f8ea57-ebcd-4755-b1f1-5fbe91ef7940 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:30:56.535868 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f8ea57-ebcd-4755-b1f1-5fbe91ef7940 admin admin] PATH_INFO: `/v3/projects/7eefe01b108148bab60d4de13be0405d` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:30:56.537007 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f8ea57-ebcd-4755-b1f1-5fbe91ef7940 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7eefe01b108148bab60d4de13be0405d)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:30:56.538976 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f8ea57-ebcd-4755-b1f1-5fbe91ef7940 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:30:56.587376 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-93f8ea57-ebcd-4755-b1f1-5fbe91ef7940 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7eefe01b108148bab60d4de13be0405d', 'request_id': 'req-93f8ea57-ebcd-4755-b1f1-5fbe91ef7940'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:30:56.595001 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-93f8ea57-ebcd-4755-b1f1-5fbe91ef7940 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7eefe01b108148bab60d4de13be0405d', 'request_id': 'req-93f8ea57-ebcd-4755-b1f1-5fbe91ef7940'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:30:56.600767 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-93f8ea57-ebcd-4755-b1f1-5fbe91ef7940 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7eefe01b108148bab60d4de13be0405d', 'request_id': 'req-93f8ea57-ebcd-4755-b1f1-5fbe91ef7940'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:30:56.611671 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-93f8ea57-ebcd-4755-b1f1-5fbe91ef7940 admin admin] The token cache is being invalidate because project 7eefe01b108148bab60d4de13be0405d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:30:56.611957 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-93f8ea57-ebcd-4755-b1f1-5fbe91ef7940 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:30:56.613317 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-93f8ea57-ebcd-4755-b1f1-5fbe91ef7940 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:30:56.615084 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1075/2163] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:30:56 2022] DELETE /identity/v3/projects/7eefe01b108148bab60d4de13be0405d => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:30:56.620019 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90905cd9-9aee-4a1f-a92e-84a58b61cc58 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:30:56.620303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90905cd9-9aee-4a1f-a92e-84a58b61cc58 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:30:56.620574 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90905cd9-9aee-4a1f-a92e-84a58b61cc58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:30:56.642197 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-90905cd9-9aee-4a1f-a92e-84a58b61cc58 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:30:56.643399 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-90905cd9-9aee-4a1f-a92e-84a58b61cc58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:30:56.707229 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1089/2164] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:30:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:30:56.715033 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522b70c5-a599-44db-9990-7c37130b7989 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:30:56.715321 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522b70c5-a599-44db-9990-7c37130b7989 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:30:56.715597 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522b70c5-a599-44db-9990-7c37130b7989 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:30:56.737172 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-522b70c5-a599-44db-9990-7c37130b7989 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:30:56.739157 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-522b70c5-a599-44db-9990-7c37130b7989 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:30:56.747151 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1076/2165] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:30:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:30:56.751424 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90905cd9-9aee-4a1f-a92e-84a58b61cc58 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:30:56.761923 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90905cd9-9aee-4a1f-a92e-84a58b61cc58 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:30:56.767414 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be0b572-07a6-44a2-9bf2-68135bcf9d19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:30:56.768343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be0b572-07a6-44a2-9bf2-68135bcf9d19 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:30:56.768343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be0b572-07a6-44a2-9bf2-68135bcf9d19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:30:56.768653 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be0b572-07a6-44a2-9bf2-68135bcf9d19 admin admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:30:56.769313 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be0b572-07a6-44a2-9bf2-68135bcf9d19 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:30:56.769666 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be0b572-07a6-44a2-9bf2-68135bcf9d19 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:30:56.771285 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be0b572-07a6-44a2-9bf2-68135bcf9d19 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:30:56.798310 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1090/2166] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:30:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:30:56.801093 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522b70c5-a599-44db-9990-7c37130b7989 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:30:56.810098 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:30:56.816762 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522b70c5-a599-44db-9990-7c37130b7989 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:30:56.823522 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-615fc261-a653-4577-92f0-d5eac441c768 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:30:56.824420 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615fc261-a653-4577-92f0-d5eac441c768 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:30:56.824950 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615fc261-a653-4577-92f0-d5eac441c768 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:30:56.825458 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615fc261-a653-4577-92f0-d5eac441c768 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:30:56.826284 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-615fc261-a653-4577-92f0-d5eac441c768 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:30:56.828416 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-615fc261-a653-4577-92f0-d5eac441c768 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:30:56.861096 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1077/2167] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:30:56 2022] POST /identity/v3/projects => generated 387 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:30:56.865934 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be0b572-07a6-44a2-9bf2-68135bcf9d19 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:30:56.875925 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be0b572-07a6-44a2-9bf2-68135bcf9d19 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:30:56.925900 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:30:56.942328 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6cd6301-9e48-4a5c-9f78-bc18beb9c605 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:30:56.943028 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6cd6301-9e48-4a5c-9f78-bc18beb9c605 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:30:56.943370 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6cd6301-9e48-4a5c-9f78-bc18beb9c605 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:30:56.943651 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6cd6301-9e48-4a5c-9f78-bc18beb9c605 admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:30:56.944301 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6cd6301-9e48-4a5c-9f78-bc18beb9c605 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:30:56.946272 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6cd6301-9e48-4a5c-9f78-bc18beb9c605 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:30:56.968366 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1091/2168] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:30:56 2022] POST /identity/v3/users => generated 294 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:30:56.972999 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-615fc261-a653-4577-92f0-d5eac441c768 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:30:56.983142 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-615fc261-a653-4577-92f0-d5eac441c768 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:30:56.988065 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc01a6f3-1875-4eb5-ab80-8a1d6507af62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:30:56.988651 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc01a6f3-1875-4eb5-ab80-8a1d6507af62 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:30:56.988929 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc01a6f3-1875-4eb5-ab80-8a1d6507af62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:30:56.989239 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc01a6f3-1875-4eb5-ab80-8a1d6507af62 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:30:56.989873 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc01a6f3-1875-4eb5-ab80-8a1d6507af62 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:30:56.991452 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc01a6f3-1875-4eb5-ab80-8a1d6507af62 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:30:57.001225 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1078/2169] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:30:56 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 27 01:30:57.005780 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6cd6301-9e48-4a5c-9f78-bc18beb9c605 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:30:57.016147 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6cd6301-9e48-4a5c-9f78-bc18beb9c605 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:30:57.022244 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7dd8fb-9d68-4ad3-8980-3258631c07ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:30:57.022843 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7dd8fb-9d68-4ad3-8980-3258631c07ed admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:30:57.023116 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7dd8fb-9d68-4ad3-8980-3258631c07ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:30:57.023386 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7dd8fb-9d68-4ad3-8980-3258631c07ed admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:30:57.024015 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f7dd8fb-9d68-4ad3-8980-3258631c07ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:30:57.025666 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f7dd8fb-9d68-4ad3-8980-3258631c07ed admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:30:57.036883 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1092/2170] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:30:57 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:30:57.041122 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc01a6f3-1875-4eb5-ab80-8a1d6507af62 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:30:57.051878 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc01a6f3-1875-4eb5-ab80-8a1d6507af62 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:30:57.057172 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338a0f69-ad2a-420b-9111-05b5ecb75841 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:30:57.057821 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338a0f69-ad2a-420b-9111-05b5ecb75841 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:30:57.058096 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338a0f69-ad2a-420b-9111-05b5ecb75841 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:30:57.058367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338a0f69-ad2a-420b-9111-05b5ecb75841 admin admin] PATH_INFO: `/v3/projects/73c30908c04242e28297b339bfcc5600/users/608f1351ab84402d8634ba99b6e26323/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:30:57.076085 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338a0f69-ad2a-420b-9111-05b5ecb75841 admin admin] RBAC: Authorizing `identity:create_grant(project_id=73c30908c04242e28297b339bfcc5600, user_id=608f1351ab84402d8634ba99b6e26323, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:30:57.082225 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338a0f69-ad2a-420b-9111-05b5ecb75841 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:30:57.099575 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1079/2171] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:30:57 2022] PUT /identity/v3/projects/73c30908c04242e28297b339bfcc5600/users/608f1351ab84402d8634ba99b6e26323/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:30:57.104674 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279d767f-e544-419c-9af6-6a3227572bfc None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:30:57.104988 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279d767f-e544-419c-9af6-6a3227572bfc None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:30:57.105300 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279d767f-e544-419c-9af6-6a3227572bfc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:30:57.125545 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-279d767f-e544-419c-9af6-6a3227572bfc None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:30:57.126817 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-279d767f-e544-419c-9af6-6a3227572bfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:30:57.199773 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1093/2172] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:30:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:30:57.209781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338a0f69-ad2a-420b-9111-05b5ecb75841 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:30:57.212645 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-338a0f69-ad2a-420b-9111-05b5ecb75841 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:30:57.278923 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338a0f69-ad2a-420b-9111-05b5ecb75841 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:30:57.282669 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd4810c-e3b1-4cab-a85d-244697ae9a2d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:30:57.283870 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd4810c-e3b1-4cab-a85d-244697ae9a2d service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:30:57.283870 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd4810c-e3b1-4cab-a85d-244697ae9a2d service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:30:57.283870 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd4810c-e3b1-4cab-a85d-244697ae9a2d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:30:57.295060 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd4810c-e3b1-4cab-a85d-244697ae9a2d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:30:57.297277 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd4810c-e3b1-4cab-a85d-244697ae9a2d service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:30:57.308831 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1080/2173] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:30:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:31:02.514985 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db9ddc4-c492-44f4-87f4-e3bbb6326d0e None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:31:02.515604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db9ddc4-c492-44f4-87f4-e3bbb6326d0e None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:31:02.515604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db9ddc4-c492-44f4-87f4-e3bbb6326d0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:31:02.584590 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-5db9ddc4-c492-44f4-87f4-e3bbb6326d0e None None] MFA Rules not processed for user `608f1351ab84402d8634ba99b6e26323`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:31:02.586342 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-5db9ddc4-c492-44f4-87f4-e3bbb6326d0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:31:02.715667 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1094/2174] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:31:02 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:31:02.734359 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd4810c-e3b1-4cab-a85d-244697ae9a2d service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:31:02.736942 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-4fd4810c-e3b1-4cab-a85d-244697ae9a2d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:31:02.863601 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd4810c-e3b1-4cab-a85d-244697ae9a2d service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:31:02.869949 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fee3383-2d44-4af4-8459-a03747d42ab8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:31:02.870565 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fee3383-2d44-4af4-8459-a03747d42ab8 service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:31:02.870838 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fee3383-2d44-4af4-8459-a03747d42ab8 service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:31:02.871105 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fee3383-2d44-4af4-8459-a03747d42ab8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:31:02.882869 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fee3383-2d44-4af4-8459-a03747d42ab8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:31:02.884488 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fee3383-2d44-4af4-8459-a03747d42ab8 service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:31:02.894818 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1081/2175] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:31:02 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:31:04.480779 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4a5ade-9710-4a6c-9b48-0fed2e5af045 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:31:04.481350 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4a5ade-9710-4a6c-9b48-0fed2e5af045 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:31:04.481350 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4a5ade-9710-4a6c-9b48-0fed2e5af045 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:31:04.504833 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-0b4a5ade-9710-4a6c-9b48-0fed2e5af045 None None] MFA Rules not processed for user `608f1351ab84402d8634ba99b6e26323`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:31:04.506316 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-0b4a5ade-9710-4a6c-9b48-0fed2e5af045 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:31:04.514387 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1095/2176] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:31:04 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:31:04.863796 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fee3383-2d44-4af4-8459-a03747d42ab8 service nova] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:31:04.865922 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-7fee3383-2d44-4af4-8459-a03747d42ab8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:31:04.878763 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fee3383-2d44-4af4-8459-a03747d42ab8 service nova] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:31:04.883140 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd21e1e2-b934-405e-8938-81f998ddaba6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:31:04.883754 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd21e1e2-b934-405e-8938-81f998ddaba6 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:31:04.884032 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd21e1e2-b934-405e-8938-81f998ddaba6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:31:04.884305 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd21e1e2-b934-405e-8938-81f998ddaba6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:31:04.886136 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-fd21e1e2-b934-405e-8938-81f998ddaba6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:31:04.896282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd21e1e2-b934-405e-8938-81f998ddaba6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:31:04.897761 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd21e1e2-b934-405e-8938-81f998ddaba6 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:31:04.906954 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1082/2177] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:31:04 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:31:14.497339 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=72,keystone:INSERT=3 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:31:14.902439 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=88,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:31:38.157184 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384bf4a9-ccd6-4c7a-b45c-e59780b8cfc4 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:31:38.158773 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384bf4a9-ccd6-4c7a-b45c-e59780b8cfc4 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:31:38.158773 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384bf4a9-ccd6-4c7a-b45c-e59780b8cfc4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:31:38.177281 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-384bf4a9-ccd6-4c7a-b45c-e59780b8cfc4 None None] MFA Rules not processed for user `99bcd22ff73b4526a54cf0e76f509436`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:31:38.178486 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-384bf4a9-ccd6-4c7a-b45c-e59780b8cfc4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:31:38.233024 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1096/2178] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:31:38 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:31:48.226343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=25 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:32:33.214556 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd21e1e2-b934-405e-8938-81f998ddaba6 service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:32:33.250330 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd21e1e2-b934-405e-8938-81f998ddaba6 service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:32:33.254294 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199a1961-c977-4375-a772-25cb13279055 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:32:33.254875 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199a1961-c977-4375-a772-25cb13279055 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:32:33.255151 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199a1961-c977-4375-a772-25cb13279055 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:32:33.255418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199a1961-c977-4375-a772-25cb13279055 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:32:33.257152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-199a1961-c977-4375-a772-25cb13279055 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:32:33.338987 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-199a1961-c977-4375-a772-25cb13279055 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:32:33.340420 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-199a1961-c977-4375-a772-25cb13279055 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:32:33.350139 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1083/2179] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:32:33 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:32:39.503777 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384bf4a9-ccd6-4c7a-b45c-e59780b8cfc4 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:32:39.516999 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384bf4a9-ccd6-4c7a-b45c-e59780b8cfc4 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:32:39.523426 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd70efd3-b0cf-4ea4-9b94-b7cc85607592 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:32:39.524037 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd70efd3-b0cf-4ea4-9b94-b7cc85607592 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:32:39.524320 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd70efd3-b0cf-4ea4-9b94-b7cc85607592 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:32:39.524593 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd70efd3-b0cf-4ea4-9b94-b7cc85607592 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:32:39.526398 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-fd70efd3-b0cf-4ea4-9b94-b7cc85607592 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:32:39.570121 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd70efd3-b0cf-4ea4-9b94-b7cc85607592 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:32:39.571567 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd70efd3-b0cf-4ea4-9b94-b7cc85607592 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:32:39.591460 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1097/2180] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:32:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:32:43.345188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=50 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:32:46.175908 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199a1961-c977-4375-a772-25cb13279055 service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:32:46.177661 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-199a1961-c977-4375-a772-25cb13279055 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:32:46.246054 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd70efd3-b0cf-4ea4-9b94-b7cc85607592 service neutron] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:32:46.247563 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-fd70efd3-b0cf-4ea4-9b94-b7cc85607592 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:32:46.259820 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd70efd3-b0cf-4ea4-9b94-b7cc85607592 service neutron] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:32:46.264708 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6238d96-2dc7-40d6-88ac-db9f23f6d942 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:32:46.265475 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6238d96-2dc7-40d6-88ac-db9f23f6d942 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:32:46.266245 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6238d96-2dc7-40d6-88ac-db9f23f6d942 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:32:46.266245 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6238d96-2dc7-40d6-88ac-db9f23f6d942 admin admin] PATH_INFO: `/v3/users/181df03f4bba4f96afc3c888b3ca9b4d` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:32:46.267468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6238d96-2dc7-40d6-88ac-db9f23f6d942 admin admin] RBAC: Authorizing `identity:delete_user(user_id=181df03f4bba4f96afc3c888b3ca9b4d)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:32:46.269979 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6238d96-2dc7-40d6-88ac-db9f23f6d942 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:32:46.297381 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199a1961-c977-4375-a772-25cb13279055 service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:32:46.303933 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6a11cb-78b0-4adc-9b67-3e477d506f4c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62051562fc0b4cd08750d2293e71787c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:32:46.304565 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6a11cb-78b0-4adc-9b67-3e477d506f4c service placement] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:32:46.304842 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6a11cb-78b0-4adc-9b67-3e477d506f4c service placement] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:32:46.305182 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6a11cb-78b0-4adc-9b67-3e477d506f4c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:32:46.306936 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-db6a11cb-78b0-4adc-9b67-3e477d506f4c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:32:46.319383 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db6a11cb-78b0-4adc-9b67-3e477d506f4c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:32:46.319542 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-b6238d96-2dc7-40d6-88ac-db9f23f6d942 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '181df03f4bba4f96afc3c888b3ca9b4d'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:32:46.321021 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db6a11cb-78b0-4adc-9b67-3e477d506f4c service placement] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:32:46.331820 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1084/2181] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:32:46 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:32:46.334329 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-b6238d96-2dc7-40d6-88ac-db9f23f6d942 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '181df03f4bba4f96afc3c888b3ca9b4d'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:32:46.347868 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-b6238d96-2dc7-40d6-88ac-db9f23f6d942 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '181df03f4bba4f96afc3c888b3ca9b4d'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:32:46.348686 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-b6238d96-2dc7-40d6-88ac-db9f23f6d942 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '181df03f4bba4f96afc3c888b3ca9b4d'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:32:46.349321 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-b6238d96-2dc7-40d6-88ac-db9f23f6d942 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '181df03f4bba4f96afc3c888b3ca9b4d'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:32:46.364490 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1098/2182] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:32:46 2022] DELETE /identity/v3/users/181df03f4bba4f96afc3c888b3ca9b4d => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:32:46.537535 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6a11cb-78b0-4adc-9b67-3e477d506f4c service placement] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:32:46.549991 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6a11cb-78b0-4adc-9b67-3e477d506f4c service placement] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:32:46.607151 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae6f0f3-575a-4912-b73b-9cb50ea4de7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:32:46.607776 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae6f0f3-575a-4912-b73b-9cb50ea4de7c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:32:46.608061 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae6f0f3-575a-4912-b73b-9cb50ea4de7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:32:46.608335 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae6f0f3-575a-4912-b73b-9cb50ea4de7c admin admin] PATH_INFO: `/v3/projects/40ecdf4e71844f7f9e3da7f59659b605` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:32:46.609489 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bae6f0f3-575a-4912-b73b-9cb50ea4de7c admin admin] RBAC: Authorizing `identity:delete_project(project_id=40ecdf4e71844f7f9e3da7f59659b605)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:32:46.611401 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bae6f0f3-575a-4912-b73b-9cb50ea4de7c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:32:46.657796 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bae6f0f3-575a-4912-b73b-9cb50ea4de7c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '40ecdf4e71844f7f9e3da7f59659b605', 'request_id': 'req-bae6f0f3-575a-4912-b73b-9cb50ea4de7c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:32:46.664896 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bae6f0f3-575a-4912-b73b-9cb50ea4de7c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '40ecdf4e71844f7f9e3da7f59659b605', 'request_id': 'req-bae6f0f3-575a-4912-b73b-9cb50ea4de7c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:32:46.670379 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bae6f0f3-575a-4912-b73b-9cb50ea4de7c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '40ecdf4e71844f7f9e3da7f59659b605', 'request_id': 'req-bae6f0f3-575a-4912-b73b-9cb50ea4de7c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:32:46.678827 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:32:46.683444 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bae6f0f3-575a-4912-b73b-9cb50ea4de7c admin admin] The token cache is being invalidate because project 40ecdf4e71844f7f9e3da7f59659b605 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:32:46.683761 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bae6f0f3-575a-4912-b73b-9cb50ea4de7c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:32:46.684480 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bae6f0f3-575a-4912-b73b-9cb50ea4de7c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:32:46.686271 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1085/2183] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:32:46 2022] DELETE /identity/v3/projects/40ecdf4e71844f7f9e3da7f59659b605 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:32:46.691741 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6238d96-2dc7-40d6-88ac-db9f23f6d942 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:32:46.693326 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-b6238d96-2dc7-40d6-88ac-db9f23f6d942 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:32:46.706054 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6238d96-2dc7-40d6-88ac-db9f23f6d942 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:32:46.761796 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be9b79a1-152b-4268-8c2c-479a69a1cee0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:32:46.762352 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9b79a1-152b-4268-8c2c-479a69a1cee0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:32:46.762628 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9b79a1-152b-4268-8c2c-479a69a1cee0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:32:46.762896 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9b79a1-152b-4268-8c2c-479a69a1cee0 admin admin] PATH_INFO: `/v3/users/9f0842753cea48fe9508de4fc4d15b5f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:32:46.764005 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be9b79a1-152b-4268-8c2c-479a69a1cee0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9f0842753cea48fe9508de4fc4d15b5f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:32:46.765949 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be9b79a1-152b-4268-8c2c-479a69a1cee0 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:32:46.787152 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:32:46.817539 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-be9b79a1-152b-4268-8c2c-479a69a1cee0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9f0842753cea48fe9508de4fc4d15b5f'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:32:46.824897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-be9b79a1-152b-4268-8c2c-479a69a1cee0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9f0842753cea48fe9508de4fc4d15b5f'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:32:46.836731 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-be9b79a1-152b-4268-8c2c-479a69a1cee0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9f0842753cea48fe9508de4fc4d15b5f'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:32:46.837370 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-be9b79a1-152b-4268-8c2c-479a69a1cee0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9f0842753cea48fe9508de4fc4d15b5f'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:32:46.837969 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-be9b79a1-152b-4268-8c2c-479a69a1cee0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9f0842753cea48fe9508de4fc4d15b5f'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:32:46.849770 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1099/2184] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:32:46 2022] DELETE /identity/v3/users/9f0842753cea48fe9508de4fc4d15b5f => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:32:47.022802 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae6f0f3-575a-4912-b73b-9cb50ea4de7c admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:32:47.024331 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-bae6f0f3-575a-4912-b73b-9cb50ea4de7c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:32:47.041528 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae6f0f3-575a-4912-b73b-9cb50ea4de7c admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:32:47.098430 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ce6d27-bc61-4470-82be-59cae299cc0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:32:47.099049 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ce6d27-bc61-4470-82be-59cae299cc0c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:32:47.099328 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ce6d27-bc61-4470-82be-59cae299cc0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:32:47.099597 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ce6d27-bc61-4470-82be-59cae299cc0c admin admin] PATH_INFO: `/v3/projects/6de71544137646399172a5df0043ebfd` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:32:47.100697 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ce6d27-bc61-4470-82be-59cae299cc0c admin admin] RBAC: Authorizing `identity:delete_project(project_id=6de71544137646399172a5df0043ebfd)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:32:47.102639 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ce6d27-bc61-4470-82be-59cae299cc0c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:32:47.150836 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-28ce6d27-bc61-4470-82be-59cae299cc0c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6de71544137646399172a5df0043ebfd', 'request_id': 'req-28ce6d27-bc61-4470-82be-59cae299cc0c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:32:47.158861 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-28ce6d27-bc61-4470-82be-59cae299cc0c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6de71544137646399172a5df0043ebfd', 'request_id': 'req-28ce6d27-bc61-4470-82be-59cae299cc0c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:32:47.165986 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-28ce6d27-bc61-4470-82be-59cae299cc0c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6de71544137646399172a5df0043ebfd', 'request_id': 'req-28ce6d27-bc61-4470-82be-59cae299cc0c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:32:47.178762 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-28ce6d27-bc61-4470-82be-59cae299cc0c admin admin] The token cache is being invalidate because project 6de71544137646399172a5df0043ebfd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:32:47.179301 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-28ce6d27-bc61-4470-82be-59cae299cc0c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:32:47.179719 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-28ce6d27-bc61-4470-82be-59cae299cc0c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:32:47.181291 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1086/2185] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:32:47 2022] DELETE /identity/v3/projects/6de71544137646399172a5df0043ebfd => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:32:56.559336 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be9b79a1-152b-4268-8c2c-479a69a1cee0 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:32:56.560968 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-be9b79a1-152b-4268-8c2c-479a69a1cee0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:32:56.633554 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be9b79a1-152b-4268-8c2c-479a69a1cee0 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:32:56.693786 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb78ebd-a862-498d-9925-73b00c74e5fe service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62051562fc0b4cd08750d2293e71787c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:32:56.694406 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb78ebd-a862-498d-9925-73b00c74e5fe service placement] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:32:56.694686 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb78ebd-a862-498d-9925-73b00c74e5fe service placement] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:32:56.694962 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb78ebd-a862-498d-9925-73b00c74e5fe service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:32:56.696706 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-6eb78ebd-a862-498d-9925-73b00c74e5fe service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:32:56.708642 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eb78ebd-a862-498d-9925-73b00c74e5fe service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:32:56.710307 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eb78ebd-a862-498d-9925-73b00c74e5fe service placement] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:32:56.720727 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1100/2186] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:32:56 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:32:57.175425 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=51,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:33:01.763705 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf1498b-2c41-4143-86f8-44e785fd2b18 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:01.764305 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf1498b-2c41-4143-86f8-44e785fd2b18 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:01.764305 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf1498b-2c41-4143-86f8-44e785fd2b18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:01.793997 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-cdf1498b-2c41-4143-86f8-44e785fd2b18 None None] MFA Rules not processed for user `7a3f8b5eb09f4ba29f4bd343bd91f6a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:33:01.795183 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-cdf1498b-2c41-4143-86f8-44e785fd2b18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:01.857553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1087/2187] 10.222.0.42 () {58 vars in 1009 bytes} [Thu Oct 27 01:33:01 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:33:01.905487 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb78ebd-a862-498d-9925-73b00c74e5fe service placement] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:01.906880 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-6eb78ebd-a862-498d-9925-73b00c74e5fe service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:01.946181 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:DELETE=7,keystone:SELECT=92,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:33:01.982602 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb78ebd-a862-498d-9925-73b00c74e5fe service placement] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:01.988479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7c183a-b7a1-4171-97db-202814c41aa2 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77f2cf9e0c3541f9bccfb7e213d88609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:01.989088 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7c183a-b7a1-4171-97db-202814c41aa2 service swift] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:01.989861 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7c183a-b7a1-4171-97db-202814c41aa2 service swift] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:01.990130 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7c183a-b7a1-4171-97db-202814c41aa2 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:01.998087 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7c183a-b7a1-4171-97db-202814c41aa2 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:01.999653 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7c183a-b7a1-4171-97db-202814c41aa2 service swift] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:02.008609 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1101/2188] 10.222.0.42 () {60 vars in 1384 bytes} [Thu Oct 27 01:33:01 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 27 01:33:02.133529 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf1498b-2c41-4143-86f8-44e785fd2b18 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:02.135105 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-cdf1498b-2c41-4143-86f8-44e785fd2b18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:02.213337 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf1498b-2c41-4143-86f8-44e785fd2b18 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:02.217790 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28bf0ca4-1857-45e3-a649-77b2a65b9f8d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:02.218405 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28bf0ca4-1857-45e3-a649-77b2a65b9f8d service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:02.218701 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28bf0ca4-1857-45e3-a649-77b2a65b9f8d service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:02.218974 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28bf0ca4-1857-45e3-a649-77b2a65b9f8d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:02.220745 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-28bf0ca4-1857-45e3-a649-77b2a65b9f8d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:02.293305 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28bf0ca4-1857-45e3-a649-77b2a65b9f8d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:02.296441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28bf0ca4-1857-45e3-a649-77b2a65b9f8d service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:02.306151 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1088/2189] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:33:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:33:06.524292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7c183a-b7a1-4171-97db-202814c41aa2 service swift] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:06.525817 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-4f7c183a-b7a1-4171-97db-202814c41aa2 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:06.548747 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7c183a-b7a1-4171-97db-202814c41aa2 service swift] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:06.555184 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841bc242-5fbc-467e-9be4-ca3426b54ed2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:06.556210 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841bc242-5fbc-467e-9be4-ca3426b54ed2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:06.556544 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841bc242-5fbc-467e-9be4-ca3426b54ed2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:06.557030 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841bc242-5fbc-467e-9be4-ca3426b54ed2 admin admin] PATH_INFO: `/v3/users/cca36a277e3643a498b27e3c5d5e14f4` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:06.558425 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841bc242-5fbc-467e-9be4-ca3426b54ed2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cca36a277e3643a498b27e3c5d5e14f4)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:06.560508 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841bc242-5fbc-467e-9be4-ca3426b54ed2 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:06.645178 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-841bc242-5fbc-467e-9be4-ca3426b54ed2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cca36a277e3643a498b27e3c5d5e14f4'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:06.657488 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-841bc242-5fbc-467e-9be4-ca3426b54ed2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cca36a277e3643a498b27e3c5d5e14f4'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:06.679814 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-841bc242-5fbc-467e-9be4-ca3426b54ed2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cca36a277e3643a498b27e3c5d5e14f4'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:06.680545 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-841bc242-5fbc-467e-9be4-ca3426b54ed2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cca36a277e3643a498b27e3c5d5e14f4'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:06.681191 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-841bc242-5fbc-467e-9be4-ca3426b54ed2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cca36a277e3643a498b27e3c5d5e14f4'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:06.699846 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1102/2190] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:33:06 2022] DELETE /identity/v3/users/cca36a277e3643a498b27e3c5d5e14f4 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:33:06.962175 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28bf0ca4-1857-45e3-a649-77b2a65b9f8d service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:06.973314 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28bf0ca4-1857-45e3-a649-77b2a65b9f8d service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:07.021108 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:33:07.062998 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0705eb-dc85-48c1-9688-e50172f55c59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:07.067165 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0705eb-dc85-48c1-9688-e50172f55c59 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:07.068052 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0705eb-dc85-48c1-9688-e50172f55c59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:07.069133 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0705eb-dc85-48c1-9688-e50172f55c59 admin admin] PATH_INFO: `/v3/projects/1b1ee6f6a3c3475e8ae993c79ace9166` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:07.071225 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0705eb-dc85-48c1-9688-e50172f55c59 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1b1ee6f6a3c3475e8ae993c79ace9166)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:07.076005 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0705eb-dc85-48c1-9688-e50172f55c59 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:07.157657 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bc0705eb-dc85-48c1-9688-e50172f55c59 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1b1ee6f6a3c3475e8ae993c79ace9166', 'request_id': 'req-bc0705eb-dc85-48c1-9688-e50172f55c59'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:07.167592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bc0705eb-dc85-48c1-9688-e50172f55c59 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1b1ee6f6a3c3475e8ae993c79ace9166', 'request_id': 'req-bc0705eb-dc85-48c1-9688-e50172f55c59'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:07.187282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bc0705eb-dc85-48c1-9688-e50172f55c59 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1b1ee6f6a3c3475e8ae993c79ace9166', 'request_id': 'req-bc0705eb-dc85-48c1-9688-e50172f55c59'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:07.218475 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bc0705eb-dc85-48c1-9688-e50172f55c59 admin admin] The token cache is being invalidate because project 1b1ee6f6a3c3475e8ae993c79ace9166 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:33:07.218773 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bc0705eb-dc85-48c1-9688-e50172f55c59 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:07.223088 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bc0705eb-dc85-48c1-9688-e50172f55c59 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:07.229490 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1089/2191] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:33:06 2022] DELETE /identity/v3/projects/1b1ee6f6a3c3475e8ae993c79ace9166 => generated 0 bytes in 264 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:33:07.231049 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7110bf-7911-4e93-8e6b-7b6de3e27c02 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:07.231322 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7110bf-7911-4e93-8e6b-7b6de3e27c02 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:07.232182 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7110bf-7911-4e93-8e6b-7b6de3e27c02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:07.325597 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-3a7110bf-7911-4e93-8e6b-7b6de3e27c02 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:33:07.327245 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-3a7110bf-7911-4e93-8e6b-7b6de3e27c02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:07.406826 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1103/2192] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:33:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:33:07.415442 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1becc860-edeb-43c7-938d-94b2655c72ea None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:07.415837 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1becc860-edeb-43c7-938d-94b2655c72ea None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:07.416218 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1becc860-edeb-43c7-938d-94b2655c72ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:07.438772 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-1becc860-edeb-43c7-938d-94b2655c72ea None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:33:07.440708 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-1becc860-edeb-43c7-938d-94b2655c72ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:07.449571 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1090/2193] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:33:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:33:07.453808 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7110bf-7911-4e93-8e6b-7b6de3e27c02 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:07.459602 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:33:07.465444 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7110bf-7911-4e93-8e6b-7b6de3e27c02 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:07.473230 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa610f1-ac60-4c72-852f-ccab5ae0f495 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:07.473230 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa610f1-ac60-4c72-852f-ccab5ae0f495 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:07.473230 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa610f1-ac60-4c72-852f-ccab5ae0f495 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:07.473608 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa610f1-ac60-4c72-852f-ccab5ae0f495 admin admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:07.474329 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa610f1-ac60-4c72-852f-ccab5ae0f495 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:33:07.474624 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa610f1-ac60-4c72-852f-ccab5ae0f495 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:07.476520 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa610f1-ac60-4c72-852f-ccab5ae0f495 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:07.491786 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1104/2194] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:33:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:33:07.496092 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1becc860-edeb-43c7-938d-94b2655c72ea None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:07.506784 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1becc860-edeb-43c7-938d-94b2655c72ea None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:07.511793 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19addef-17fb-4786-bacd-76b28f19a153 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:07.512483 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19addef-17fb-4786-bacd-76b28f19a153 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:07.512813 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19addef-17fb-4786-bacd-76b28f19a153 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:07.513135 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19addef-17fb-4786-bacd-76b28f19a153 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:07.513789 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19addef-17fb-4786-bacd-76b28f19a153 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:07.515728 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19addef-17fb-4786-bacd-76b28f19a153 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:07.546966 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1091/2195] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:33:07 2022] POST /identity/v3/projects => generated 369 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:33:07.551957 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa610f1-ac60-4c72-852f-ccab5ae0f495 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:07.564594 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa610f1-ac60-4c72-852f-ccab5ae0f495 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:07.634629 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b9e867c-ed27-4d1a-af8e-7abfee93f850 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:07.635141 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9e867c-ed27-4d1a-af8e-7abfee93f850 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:07.635416 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9e867c-ed27-4d1a-af8e-7abfee93f850 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:07.635680 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9e867c-ed27-4d1a-af8e-7abfee93f850 admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:07.636307 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b9e867c-ed27-4d1a-af8e-7abfee93f850 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:07.639079 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b9e867c-ed27-4d1a-af8e-7abfee93f850 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:07.669717 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1105/2196] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:33:07 2022] POST /identity/v3/users => generated 285 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:33:07.670880 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19addef-17fb-4786-bacd-76b28f19a153 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:07.683484 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19addef-17fb-4786-bacd-76b28f19a153 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:07.688639 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fad98cb-0c0b-4316-b3e8-0d23ff72fbee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:07.689402 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fad98cb-0c0b-4316-b3e8-0d23ff72fbee admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:07.690327 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fad98cb-0c0b-4316-b3e8-0d23ff72fbee admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:07.690595 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fad98cb-0c0b-4316-b3e8-0d23ff72fbee admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:07.691240 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fad98cb-0c0b-4316-b3e8-0d23ff72fbee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:07.692833 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fad98cb-0c0b-4316-b3e8-0d23ff72fbee admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:07.706380 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1092/2197] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:33:07 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 27 01:33:07.707707 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b9e867c-ed27-4d1a-af8e-7abfee93f850 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:07.718707 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b9e867c-ed27-4d1a-af8e-7abfee93f850 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:07.724496 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3428e0d2-1556-4251-80eb-076d72d43ca3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:07.725640 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3428e0d2-1556-4251-80eb-076d72d43ca3 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:07.726169 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3428e0d2-1556-4251-80eb-076d72d43ca3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:07.726677 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3428e0d2-1556-4251-80eb-076d72d43ca3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:07.727472 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3428e0d2-1556-4251-80eb-076d72d43ca3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:07.729186 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3428e0d2-1556-4251-80eb-076d72d43ca3 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:07.748393 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1106/2198] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:33:07 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:33:07.754198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fad98cb-0c0b-4316-b3e8-0d23ff72fbee admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:07.766448 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fad98cb-0c0b-4316-b3e8-0d23ff72fbee admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:07.772584 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f970ff6f-a745-4ee3-ac3c-23958433532b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:07.773550 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f970ff6f-a745-4ee3-ac3c-23958433532b admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:07.774632 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f970ff6f-a745-4ee3-ac3c-23958433532b admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:07.775077 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f970ff6f-a745-4ee3-ac3c-23958433532b admin admin] PATH_INFO: `/v3/projects/a2f507e377d945a2b332640ad2349451/users/a49d35295bba4e81b164e1946781aaa4/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:07.792783 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f970ff6f-a745-4ee3-ac3c-23958433532b admin admin] RBAC: Authorizing `identity:create_grant(project_id=a2f507e377d945a2b332640ad2349451, user_id=a49d35295bba4e81b164e1946781aaa4, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:07.795479 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f970ff6f-a745-4ee3-ac3c-23958433532b admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:07.818683 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1093/2199] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:33:07 2022] PUT /identity/v3/projects/a2f507e377d945a2b332640ad2349451/users/a49d35295bba4e81b164e1946781aaa4/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:33:07.828746 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c419afd-0e97-44dd-b97a-f046e13c3212 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:07.829037 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c419afd-0e97-44dd-b97a-f046e13c3212 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:07.829341 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c419afd-0e97-44dd-b97a-f046e13c3212 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:07.866320 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-1c419afd-0e97-44dd-b97a-f046e13c3212 None None] MFA Rules not processed for user `a49d35295bba4e81b164e1946781aaa4`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:33:07.867758 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-1c419afd-0e97-44dd-b97a-f046e13c3212 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:07.965220 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1107/2200] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:33:07 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:33:07.966475 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f970ff6f-a745-4ee3-ac3c-23958433532b admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:08.041849 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:33:08.047044 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f970ff6f-a745-4ee3-ac3c-23958433532b admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:08.057537 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1931330-0c0d-45ef-900e-9c1dae7784ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:08.058173 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1931330-0c0d-45ef-900e-9c1dae7784ec admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:08.058320 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1931330-0c0d-45ef-900e-9c1dae7784ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:08.058586 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1931330-0c0d-45ef-900e-9c1dae7784ec admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:08.059745 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1931330-0c0d-45ef-900e-9c1dae7784ec admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:08.062411 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1931330-0c0d-45ef-900e-9c1dae7784ec admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:08.098867 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1094/2201] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:33:07 2022] POST /identity/v3/projects => generated 369 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:33:08.103277 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c419afd-0e97-44dd-b97a-f046e13c3212 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:08.114787 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c419afd-0e97-44dd-b97a-f046e13c3212 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:08.178293 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:33:08.185396 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14116b00-3bdf-407f-bf00-612eee01e082 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:08.186448 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14116b00-3bdf-407f-bf00-612eee01e082 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:08.186730 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14116b00-3bdf-407f-bf00-612eee01e082 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:08.186999 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14116b00-3bdf-407f-bf00-612eee01e082 admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:08.187777 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14116b00-3bdf-407f-bf00-612eee01e082 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:08.191213 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14116b00-3bdf-407f-bf00-612eee01e082 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:08.217345 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1108/2202] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:33:08 2022] POST /identity/v3/users => generated 291 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:33:08.221205 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1931330-0c0d-45ef-900e-9c1dae7784ec admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:08.231874 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1931330-0c0d-45ef-900e-9c1dae7784ec admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:08.236659 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b2d5ed-e925-446a-8e55-1ebe89ed2b87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:08.237317 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b2d5ed-e925-446a-8e55-1ebe89ed2b87 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:08.237591 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b2d5ed-e925-446a-8e55-1ebe89ed2b87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:08.237864 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b2d5ed-e925-446a-8e55-1ebe89ed2b87 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:08.238753 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b2d5ed-e925-446a-8e55-1ebe89ed2b87 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:08.240415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b2d5ed-e925-446a-8e55-1ebe89ed2b87 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:08.252962 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1095/2203] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:33:08 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:33:08.257325 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14116b00-3bdf-407f-bf00-612eee01e082 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:08.268846 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14116b00-3bdf-407f-bf00-612eee01e082 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:08.273776 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c55f0c-aa84-4d39-8367-c1a50328172c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:08.274533 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c55f0c-aa84-4d39-8367-c1a50328172c admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:08.274910 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c55f0c-aa84-4d39-8367-c1a50328172c admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:08.275299 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c55f0c-aa84-4d39-8367-c1a50328172c admin admin] PATH_INFO: `/v3/projects/e2b279a82d384c9b93675d9fd575c8dc/users/620d59385033468bbfa44ad7b0bc0d4e/roles/414286927fdd44688107ff5e10235acf` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:08.293831 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c55f0c-aa84-4d39-8367-c1a50328172c admin admin] RBAC: Authorizing `identity:create_grant(project_id=e2b279a82d384c9b93675d9fd575c8dc, user_id=620d59385033468bbfa44ad7b0bc0d4e, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:08.296647 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c55f0c-aa84-4d39-8367-c1a50328172c admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:08.320579 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1109/2204] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:33:08 2022] PUT /identity/v3/projects/e2b279a82d384c9b93675d9fd575c8dc/users/620d59385033468bbfa44ad7b0bc0d4e/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:33:08.321963 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b2d5ed-e925-446a-8e55-1ebe89ed2b87 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:08.334915 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b2d5ed-e925-446a-8e55-1ebe89ed2b87 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:08.407736 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0215055b-cc12-4898-b42a-cc85296a9042 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:08.408936 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0215055b-cc12-4898-b42a-cc85296a9042 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:08.409255 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0215055b-cc12-4898-b42a-cc85296a9042 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:08.409529 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0215055b-cc12-4898-b42a-cc85296a9042 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:08.410181 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0215055b-cc12-4898-b42a-cc85296a9042 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:08.411800 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0215055b-cc12-4898-b42a-cc85296a9042 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:08.427683 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1096/2205] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:33:08 2022] GET /identity/v3/roles => generated 1420 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:33:08.428921 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c55f0c-aa84-4d39-8367-c1a50328172c admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:08.451208 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c55f0c-aa84-4d39-8367-c1a50328172c admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:08.458124 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08cc2ae3-a1e8-4dc3-bb71-4c240cb1471c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:08.459003 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cc2ae3-a1e8-4dc3-bb71-4c240cb1471c admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:08.459343 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cc2ae3-a1e8-4dc3-bb71-4c240cb1471c admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:08.459683 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cc2ae3-a1e8-4dc3-bb71-4c240cb1471c admin admin] PATH_INFO: `/v3/projects/e2b279a82d384c9b93675d9fd575c8dc/users/620d59385033468bbfa44ad7b0bc0d4e/roles/414286927fdd44688107ff5e10235acf` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:08.462213 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08cc2ae3-a1e8-4dc3-bb71-4c240cb1471c admin admin] RBAC: Authorizing `identity:create_grant(project_id=e2b279a82d384c9b93675d9fd575c8dc, user_id=620d59385033468bbfa44ad7b0bc0d4e, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:08.465142 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08cc2ae3-a1e8-4dc3-bb71-4c240cb1471c admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:08.485793 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1110/2206] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:33:08 2022] PUT /identity/v3/projects/e2b279a82d384c9b93675d9fd575c8dc/users/620d59385033468bbfa44ad7b0bc0d4e/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:33:08.487659 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34972b85-d954-46dd-8d32-8d6513bd6279 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:08.488344 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34972b85-d954-46dd-8d32-8d6513bd6279 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:08.488778 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34972b85-d954-46dd-8d32-8d6513bd6279 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:08.528141 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-34972b85-d954-46dd-8d32-8d6513bd6279 None None] MFA Rules not processed for user `620d59385033468bbfa44ad7b0bc0d4e`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:33:08.529429 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-34972b85-d954-46dd-8d32-8d6513bd6279 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:08.629674 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1097/2207] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:33:08 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:33:08.640367 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08cc2ae3-a1e8-4dc3-bb71-4c240cb1471c admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:08.642252 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-08cc2ae3-a1e8-4dc3-bb71-4c240cb1471c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:08.764633 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08cc2ae3-a1e8-4dc3-bb71-4c240cb1471c admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:08.769480 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4353194c-150b-40c5-9877-6a3e55d6649e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:08.770232 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4353194c-150b-40c5-9877-6a3e55d6649e service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:08.770576 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4353194c-150b-40c5-9877-6a3e55d6649e service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:08.770934 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4353194c-150b-40c5-9877-6a3e55d6649e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:08.806931 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4353194c-150b-40c5-9877-6a3e55d6649e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:08.808503 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4353194c-150b-40c5-9877-6a3e55d6649e service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:08.838167 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1111/2208] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:33:08 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 198 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:33:10.672501 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34972b85-d954-46dd-8d32-8d6513bd6279 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:10.737151 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:33:10.758046 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34972b85-d954-46dd-8d32-8d6513bd6279 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:10.764343 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdce5fd4-2c72-4859-a4f3-0318e40fef6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:10.765007 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdce5fd4-2c72-4859-a4f3-0318e40fef6c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:10.765386 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdce5fd4-2c72-4859-a4f3-0318e40fef6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:10.765834 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdce5fd4-2c72-4859-a4f3-0318e40fef6c admin admin] PATH_INFO: `/v3/users/a49d35295bba4e81b164e1946781aaa4` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:10.767310 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdce5fd4-2c72-4859-a4f3-0318e40fef6c admin admin] RBAC: Authorizing `identity:delete_user(user_id=a49d35295bba4e81b164e1946781aaa4)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:10.770804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdce5fd4-2c72-4859-a4f3-0318e40fef6c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:10.842331 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bdce5fd4-2c72-4859-a4f3-0318e40fef6c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a49d35295bba4e81b164e1946781aaa4'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:10.856020 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bdce5fd4-2c72-4859-a4f3-0318e40fef6c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a49d35295bba4e81b164e1946781aaa4'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:10.857471 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bdce5fd4-2c72-4859-a4f3-0318e40fef6c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a49d35295bba4e81b164e1946781aaa4'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:10.872293 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bdce5fd4-2c72-4859-a4f3-0318e40fef6c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a49d35295bba4e81b164e1946781aaa4'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:10.884415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bdce5fd4-2c72-4859-a4f3-0318e40fef6c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a49d35295bba4e81b164e1946781aaa4'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:10.890901 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1098/2209] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:33:10 2022] DELETE /identity/v3/users/a49d35295bba4e81b164e1946781aaa4 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:33:10.896487 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d240f69b-af21-4618-a46d-9bea35cec005 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:10.896988 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d240f69b-af21-4618-a46d-9bea35cec005 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:10.897346 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d240f69b-af21-4618-a46d-9bea35cec005 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:10.922807 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-d240f69b-af21-4618-a46d-9bea35cec005 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:33:10.924954 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-d240f69b-af21-4618-a46d-9bea35cec005 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:11.021051 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1112/2210] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:33:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:33:11.040284 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdce5fd4-2c72-4859-a4f3-0318e40fef6c admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:11.041972 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-bdce5fd4-2c72-4859-a4f3-0318e40fef6c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:11.112425 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdce5fd4-2c72-4859-a4f3-0318e40fef6c admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:11.116583 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8385a5b7-6873-44cc-ae5b-131d6f42a70b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:11.117299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8385a5b7-6873-44cc-ae5b-131d6f42a70b service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:11.117580 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8385a5b7-6873-44cc-ae5b-131d6f42a70b service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:11.117848 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8385a5b7-6873-44cc-ae5b-131d6f42a70b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:11.129621 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8385a5b7-6873-44cc-ae5b-131d6f42a70b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:11.131642 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8385a5b7-6873-44cc-ae5b-131d6f42a70b service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:11.145261 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1099/2211] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:33:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:33:11.458862 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d240f69b-af21-4618-a46d-9bea35cec005 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:11.460546 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-d240f69b-af21-4618-a46d-9bea35cec005 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:11.475244 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d240f69b-af21-4618-a46d-9bea35cec005 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:11.479986 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2978387-4d95-49ea-bb31-984a3299eab6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:11.480626 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2978387-4d95-49ea-bb31-984a3299eab6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:11.480921 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2978387-4d95-49ea-bb31-984a3299eab6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:11.481203 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2978387-4d95-49ea-bb31-984a3299eab6 admin admin] PATH_INFO: `/v3/projects/a2f507e377d945a2b332640ad2349451` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:11.482413 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2978387-4d95-49ea-bb31-984a3299eab6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a2f507e377d945a2b332640ad2349451)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:11.484603 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2978387-4d95-49ea-bb31-984a3299eab6 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:11.526281 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=93,keystone:INSERT=5,keystone:DELETE=2 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:33:11.550306 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-b2978387-4d95-49ea-bb31-984a3299eab6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a2f507e377d945a2b332640ad2349451', 'request_id': 'req-b2978387-4d95-49ea-bb31-984a3299eab6'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:11.568052 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-b2978387-4d95-49ea-bb31-984a3299eab6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a2f507e377d945a2b332640ad2349451', 'request_id': 'req-b2978387-4d95-49ea-bb31-984a3299eab6'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:11.578295 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-b2978387-4d95-49ea-bb31-984a3299eab6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a2f507e377d945a2b332640ad2349451', 'request_id': 'req-b2978387-4d95-49ea-bb31-984a3299eab6'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:11.593527 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-b2978387-4d95-49ea-bb31-984a3299eab6 admin admin] The token cache is being invalidate because project a2f507e377d945a2b332640ad2349451 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:33:11.593960 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-b2978387-4d95-49ea-bb31-984a3299eab6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:11.594672 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-b2978387-4d95-49ea-bb31-984a3299eab6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:11.596574 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1113/2212] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:33:11 2022] DELETE /identity/v3/projects/a2f507e377d945a2b332640ad2349451 => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:33:11.600957 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8385a5b7-6873-44cc-ae5b-131d6f42a70b service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:11.612606 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8385a5b7-6873-44cc-ae5b-131d6f42a70b service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:11.678700 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415f9e7b-c4bb-41d3-9b53-d578d6e2c088 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:11.680945 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415f9e7b-c4bb-41d3-9b53-d578d6e2c088 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:11.681473 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415f9e7b-c4bb-41d3-9b53-d578d6e2c088 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:11.681894 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415f9e7b-c4bb-41d3-9b53-d578d6e2c088 admin admin] PATH_INFO: `/v3/users/620d59385033468bbfa44ad7b0bc0d4e` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:11.683263 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415f9e7b-c4bb-41d3-9b53-d578d6e2c088 admin admin] RBAC: Authorizing `identity:delete_user(user_id=620d59385033468bbfa44ad7b0bc0d4e)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:11.685406 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415f9e7b-c4bb-41d3-9b53-d578d6e2c088 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:11.704741 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:33:11.760462 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-415f9e7b-c4bb-41d3-9b53-d578d6e2c088 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '620d59385033468bbfa44ad7b0bc0d4e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:11.778648 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-415f9e7b-c4bb-41d3-9b53-d578d6e2c088 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '620d59385033468bbfa44ad7b0bc0d4e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:11.779283 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-415f9e7b-c4bb-41d3-9b53-d578d6e2c088 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '620d59385033468bbfa44ad7b0bc0d4e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:11.793827 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-415f9e7b-c4bb-41d3-9b53-d578d6e2c088 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '620d59385033468bbfa44ad7b0bc0d4e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:11.803236 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-415f9e7b-c4bb-41d3-9b53-d578d6e2c088 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '620d59385033468bbfa44ad7b0bc0d4e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:11.810055 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1100/2213] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:33:11 2022] DELETE /identity/v3/users/620d59385033468bbfa44ad7b0bc0d4e => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:33:12.155188 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2978387-4d95-49ea-bb31-984a3299eab6 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:12.156865 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-b2978387-4d95-49ea-bb31-984a3299eab6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:12.177598 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2978387-4d95-49ea-bb31-984a3299eab6 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:12.243846 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093495f1-9d6e-4fee-ac32-bdc305ba212e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:12.244388 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093495f1-9d6e-4fee-ac32-bdc305ba212e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:12.244664 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093495f1-9d6e-4fee-ac32-bdc305ba212e admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:12.244928 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093495f1-9d6e-4fee-ac32-bdc305ba212e admin admin] PATH_INFO: `/v3/projects/e2b279a82d384c9b93675d9fd575c8dc` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:12.246071 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-093495f1-9d6e-4fee-ac32-bdc305ba212e admin admin] RBAC: Authorizing `identity:delete_project(project_id=e2b279a82d384c9b93675d9fd575c8dc)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:12.247977 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-093495f1-9d6e-4fee-ac32-bdc305ba212e admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:12.301637 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-093495f1-9d6e-4fee-ac32-bdc305ba212e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e2b279a82d384c9b93675d9fd575c8dc', 'request_id': 'req-093495f1-9d6e-4fee-ac32-bdc305ba212e'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:12.312522 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-093495f1-9d6e-4fee-ac32-bdc305ba212e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e2b279a82d384c9b93675d9fd575c8dc', 'request_id': 'req-093495f1-9d6e-4fee-ac32-bdc305ba212e'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:12.320173 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-093495f1-9d6e-4fee-ac32-bdc305ba212e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e2b279a82d384c9b93675d9fd575c8dc', 'request_id': 'req-093495f1-9d6e-4fee-ac32-bdc305ba212e'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:12.334907 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-093495f1-9d6e-4fee-ac32-bdc305ba212e admin admin] The token cache is being invalidate because project e2b279a82d384c9b93675d9fd575c8dc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:33:12.335282 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-093495f1-9d6e-4fee-ac32-bdc305ba212e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:12.336005 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-093495f1-9d6e-4fee-ac32-bdc305ba212e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:12.337591 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1114/2214] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:33:12 2022] DELETE /identity/v3/projects/e2b279a82d384c9b93675d9fd575c8dc => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:33:12.342535 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d104535-9c6c-40bb-952a-116bd15f98f3 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:12.342887 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d104535-9c6c-40bb-952a-116bd15f98f3 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:12.343229 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d104535-9c6c-40bb-952a-116bd15f98f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:12.365604 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-9d104535-9c6c-40bb-952a-116bd15f98f3 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:33:12.367125 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-9d104535-9c6c-40bb-952a-116bd15f98f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:12.437707 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1101/2215] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:33:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:33:12.446068 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01927e48-3ce4-4832-8940-2170e4cb563f None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:12.446371 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01927e48-3ce4-4832-8940-2170e4cb563f None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:12.446640 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01927e48-3ce4-4832-8940-2170e4cb563f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:12.467306 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-01927e48-3ce4-4832-8940-2170e4cb563f None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:33:12.468504 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-01927e48-3ce4-4832-8940-2170e4cb563f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:12.476865 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1115/2216] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:33:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:33:12.482077 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d104535-9c6c-40bb-952a-116bd15f98f3 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:12.493824 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d104535-9c6c-40bb-952a-116bd15f98f3 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:12.500182 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96f6eb3-a60b-4be0-ac7c-e547a8ac0832 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:12.500734 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96f6eb3-a60b-4be0-ac7c-e547a8ac0832 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:12.501178 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96f6eb3-a60b-4be0-ac7c-e547a8ac0832 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:12.501454 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96f6eb3-a60b-4be0-ac7c-e547a8ac0832 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:12.502851 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d96f6eb3-a60b-4be0-ac7c-e547a8ac0832 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:33:12.503324 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d96f6eb3-a60b-4be0-ac7c-e547a8ac0832 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:12.504929 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d96f6eb3-a60b-4be0-ac7c-e547a8ac0832 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:12.517592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1102/2217] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:33:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:33:12.522315 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01927e48-3ce4-4832-8940-2170e4cb563f None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:12.531735 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01927e48-3ce4-4832-8940-2170e4cb563f None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:12.537576 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af726aa4-d3d0-4e8e-bdbf-9dd9dd37fd1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:12.538277 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af726aa4-d3d0-4e8e-bdbf-9dd9dd37fd1c admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:12.538609 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af726aa4-d3d0-4e8e-bdbf-9dd9dd37fd1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:12.538947 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af726aa4-d3d0-4e8e-bdbf-9dd9dd37fd1c admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:12.539660 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af726aa4-d3d0-4e8e-bdbf-9dd9dd37fd1c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:12.541719 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af726aa4-d3d0-4e8e-bdbf-9dd9dd37fd1c admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:12.578362 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1116/2218] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:33:12 2022] POST /identity/v3/projects => generated 401 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:33:12.584022 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96f6eb3-a60b-4be0-ac7c-e547a8ac0832 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:12.595177 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96f6eb3-a60b-4be0-ac7c-e547a8ac0832 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:12.665812 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddbc86b-c89a-4a1d-b827-16e591d1cb3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:12.666565 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddbc86b-c89a-4a1d-b827-16e591d1cb3a admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:12.666840 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddbc86b-c89a-4a1d-b827-16e591d1cb3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:12.667113 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddbc86b-c89a-4a1d-b827-16e591d1cb3a admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:12.667768 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cddbc86b-c89a-4a1d-b827-16e591d1cb3a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:12.669734 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cddbc86b-c89a-4a1d-b827-16e591d1cb3a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:12.701598 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1103/2219] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:33:12 2022] POST /identity/v3/users => generated 301 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:33:12.703123 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af726aa4-d3d0-4e8e-bdbf-9dd9dd37fd1c admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:12.714670 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af726aa4-d3d0-4e8e-bdbf-9dd9dd37fd1c admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:12.719902 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5066d6f4-a660-4c7d-9fff-ac21676e0d13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:12.721128 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5066d6f4-a660-4c7d-9fff-ac21676e0d13 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:12.721515 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5066d6f4-a660-4c7d-9fff-ac21676e0d13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:12.721783 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5066d6f4-a660-4c7d-9fff-ac21676e0d13 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:12.722424 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5066d6f4-a660-4c7d-9fff-ac21676e0d13 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:12.723999 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5066d6f4-a660-4c7d-9fff-ac21676e0d13 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:12.737829 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1117/2220] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:33:12 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:33:12.739010 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddbc86b-c89a-4a1d-b827-16e591d1cb3a admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:12.749099 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddbc86b-c89a-4a1d-b827-16e591d1cb3a admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:12.754005 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7c6fb0-0fd5-4cf5-8d6f-f6f5ac70273a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:12.754658 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7c6fb0-0fd5-4cf5-8d6f-f6f5ac70273a admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:12.755009 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7c6fb0-0fd5-4cf5-8d6f-f6f5ac70273a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:12.755277 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7c6fb0-0fd5-4cf5-8d6f-f6f5ac70273a admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:12.755915 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7c6fb0-0fd5-4cf5-8d6f-f6f5ac70273a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:12.757530 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7c6fb0-0fd5-4cf5-8d6f-f6f5ac70273a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:12.759763 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=87,keystone:DELETE=9,keystone:INSERT=4 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:33:12.771466 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1104/2221] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:33:12 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:33:12.775460 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5066d6f4-a660-4c7d-9fff-ac21676e0d13 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:12.786842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5066d6f4-a660-4c7d-9fff-ac21676e0d13 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:12.792625 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da54bdbb-ef5c-4d11-9ebd-8e1046f9dc45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:12.793495 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da54bdbb-ef5c-4d11-9ebd-8e1046f9dc45 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:12.793987 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da54bdbb-ef5c-4d11-9ebd-8e1046f9dc45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:12.794430 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da54bdbb-ef5c-4d11-9ebd-8e1046f9dc45 admin admin] PATH_INFO: `/v3/projects/d64549dafdd9476fa6848f15c48e12e9/users/583c0cdd199e47cdaa664b9e891472ae/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:12.797651 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:33:12.820427 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da54bdbb-ef5c-4d11-9ebd-8e1046f9dc45 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d64549dafdd9476fa6848f15c48e12e9, user_id=583c0cdd199e47cdaa664b9e891472ae, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:12.823640 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da54bdbb-ef5c-4d11-9ebd-8e1046f9dc45 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:12.842881 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1118/2222] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:33:12 2022] PUT /identity/v3/projects/d64549dafdd9476fa6848f15c48e12e9/users/583c0cdd199e47cdaa664b9e891472ae/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:33:12.853976 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ee7f72-298b-4e55-a9f7-99d262b5d1a1 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:12.854253 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ee7f72-298b-4e55-a9f7-99d262b5d1a1 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:12.854542 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ee7f72-298b-4e55-a9f7-99d262b5d1a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:12.896805 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-84ee7f72-298b-4e55-a9f7-99d262b5d1a1 None None] MFA Rules not processed for user `583c0cdd199e47cdaa664b9e891472ae`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:33:12.898217 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-84ee7f72-298b-4e55-a9f7-99d262b5d1a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:12.993094 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1105/2223] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:33:12 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:33:12.998416 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da54bdbb-ef5c-4d11-9ebd-8e1046f9dc45 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:13.079485 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da54bdbb-ef5c-4d11-9ebd-8e1046f9dc45 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:13.084617 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b368b69a-df97-4db9-8c68-84bba9671ef2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:13.085255 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b368b69a-df97-4db9-8c68-84bba9671ef2 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:13.085535 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b368b69a-df97-4db9-8c68-84bba9671ef2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:13.085805 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b368b69a-df97-4db9-8c68-84bba9671ef2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:13.086429 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b368b69a-df97-4db9-8c68-84bba9671ef2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:13.088325 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b368b69a-df97-4db9-8c68-84bba9671ef2 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:13.114915 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1119/2224] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:33:12 2022] POST /identity/v3/projects => generated 401 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:33:13.122213 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ee7f72-298b-4e55-a9f7-99d262b5d1a1 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:13.133577 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ee7f72-298b-4e55-a9f7-99d262b5d1a1 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:13.191179 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105011b7-c1f8-4336-b8e5-1295d8873699 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:13.191787 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105011b7-c1f8-4336-b8e5-1295d8873699 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:13.192050 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105011b7-c1f8-4336-b8e5-1295d8873699 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:13.192320 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105011b7-c1f8-4336-b8e5-1295d8873699 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:13.192943 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-105011b7-c1f8-4336-b8e5-1295d8873699 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:13.194895 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-105011b7-c1f8-4336-b8e5-1295d8873699 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:13.216763 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1106/2225] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:33:13 2022] POST /identity/v3/users => generated 307 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:33:13.221583 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b368b69a-df97-4db9-8c68-84bba9671ef2 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:13.231808 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b368b69a-df97-4db9-8c68-84bba9671ef2 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:13.236447 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9697c61-982e-4521-9922-15c2bb318e49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:13.237095 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9697c61-982e-4521-9922-15c2bb318e49 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:13.237377 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9697c61-982e-4521-9922-15c2bb318e49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:13.237644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9697c61-982e-4521-9922-15c2bb318e49 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:13.238277 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9697c61-982e-4521-9922-15c2bb318e49 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:13.239863 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9697c61-982e-4521-9922-15c2bb318e49 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:13.250137 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1120/2226] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:33:13 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:33:13.254159 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105011b7-c1f8-4336-b8e5-1295d8873699 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:13.263786 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105011b7-c1f8-4336-b8e5-1295d8873699 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:13.269029 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771404d7-9fe3-406e-8549-e96dc7672ed6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:13.269642 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771404d7-9fe3-406e-8549-e96dc7672ed6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:13.269917 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771404d7-9fe3-406e-8549-e96dc7672ed6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:13.270190 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771404d7-9fe3-406e-8549-e96dc7672ed6 admin admin] PATH_INFO: `/v3/projects/39f4415f672f45928fb4d2719f7c4ccd/users/e4ebdf87ba354977b89a50092027e10e/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:13.285259 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-771404d7-9fe3-406e-8549-e96dc7672ed6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=39f4415f672f45928fb4d2719f7c4ccd, user_id=e4ebdf87ba354977b89a50092027e10e, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:13.287902 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-771404d7-9fe3-406e-8549-e96dc7672ed6 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:13.300711 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1107/2227] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:33:13 2022] PUT /identity/v3/projects/39f4415f672f45928fb4d2719f7c4ccd/users/e4ebdf87ba354977b89a50092027e10e/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:33:13.304588 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9697c61-982e-4521-9922-15c2bb318e49 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:13.315069 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9697c61-982e-4521-9922-15c2bb318e49 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:13.369895 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3f3124-ca05-4912-91a0-97ad1b514359 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:13.370489 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3f3124-ca05-4912-91a0-97ad1b514359 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:13.370768 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3f3124-ca05-4912-91a0-97ad1b514359 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:13.371100 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3f3124-ca05-4912-91a0-97ad1b514359 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:13.371649 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3f3124-ca05-4912-91a0-97ad1b514359 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:13.373259 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3f3124-ca05-4912-91a0-97ad1b514359 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:13.382769 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1121/2228] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:33:13 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:33:13.387082 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771404d7-9fe3-406e-8549-e96dc7672ed6 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:13.397429 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771404d7-9fe3-406e-8549-e96dc7672ed6 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:13.401976 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25aaca43-dffc-48f3-bc70-d75d94e89a0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:13.402612 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25aaca43-dffc-48f3-bc70-d75d94e89a0a admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:13.402887 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25aaca43-dffc-48f3-bc70-d75d94e89a0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:13.403221 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25aaca43-dffc-48f3-bc70-d75d94e89a0a admin admin] PATH_INFO: `/v3/projects/39f4415f672f45928fb4d2719f7c4ccd/users/e4ebdf87ba354977b89a50092027e10e/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:13.405164 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25aaca43-dffc-48f3-bc70-d75d94e89a0a admin admin] RBAC: Authorizing `identity:create_grant(project_id=39f4415f672f45928fb4d2719f7c4ccd, user_id=e4ebdf87ba354977b89a50092027e10e, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:13.407882 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25aaca43-dffc-48f3-bc70-d75d94e89a0a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:13.418882 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1108/2229] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:33:13 2022] PUT /identity/v3/projects/39f4415f672f45928fb4d2719f7c4ccd/users/e4ebdf87ba354977b89a50092027e10e/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:33:13.431352 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813d48f4-15fb-4359-8d58-dc0a7f73af7d None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:13.431662 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813d48f4-15fb-4359-8d58-dc0a7f73af7d None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:13.431937 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813d48f4-15fb-4359-8d58-dc0a7f73af7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:13.460349 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-813d48f4-15fb-4359-8d58-dc0a7f73af7d None None] MFA Rules not processed for user `e4ebdf87ba354977b89a50092027e10e`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:33:13.461568 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-813d48f4-15fb-4359-8d58-dc0a7f73af7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:13.502642 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:33:13.528903 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1122/2230] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:33:13 2022] POST /identity/v3/auth/tokens => generated 3345 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:33:13.540024 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25aaca43-dffc-48f3-bc70-d75d94e89a0a admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:13.541717 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-25aaca43-dffc-48f3-bc70-d75d94e89a0a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:13.586432 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:33:13.609499 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25aaca43-dffc-48f3-bc70-d75d94e89a0a admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:13.613733 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73fe03f0-0ce3-4d30-b76c-abf199e8dbbe service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:13.614331 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73fe03f0-0ce3-4d30-b76c-abf199e8dbbe service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:13.614609 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73fe03f0-0ce3-4d30-b76c-abf199e8dbbe service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:13.614880 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73fe03f0-0ce3-4d30-b76c-abf199e8dbbe service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:13.623906 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73fe03f0-0ce3-4d30-b76c-abf199e8dbbe service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:13.625392 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73fe03f0-0ce3-4d30-b76c-abf199e8dbbe service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:13.634683 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1109/2231] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:33:13 2022] GET /identity/v3/auth/tokens => generated 3345 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:33:13.710419 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-813d48f4-15fb-4359-8d58-dc0a7f73af7d None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:13.719907 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-813d48f4-15fb-4359-8d58-dc0a7f73af7d None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:13.724104 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7824723a-6909-4108-911d-6355ee22f673 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:13.724691 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7824723a-6909-4108-911d-6355ee22f673 service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:13.724968 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7824723a-6909-4108-911d-6355ee22f673 service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:13.725264 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7824723a-6909-4108-911d-6355ee22f673 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:13.781196 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7824723a-6909-4108-911d-6355ee22f673 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:13.782551 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7824723a-6909-4108-911d-6355ee22f673 service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:13.792173 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1123/2232] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:33:13 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:33:17.327781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd33b239-025c-421e-982a-6f798f575a5c None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:17.328301 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd33b239-025c-421e-982a-6f798f575a5c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:17.328301 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd33b239-025c-421e-982a-6f798f575a5c None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:17.329675 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1110/2233] 10.222.0.42 () {54 vars in 921 bytes} [Thu Oct 27 01:33:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:33:17.334757 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fddc781-cf24-43a9-8278-03f180c5242f None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:17.335192 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fddc781-cf24-43a9-8278-03f180c5242f None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:17.335292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fddc781-cf24-43a9-8278-03f180c5242f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:17.372642 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-4fddc781-cf24-43a9-8278-03f180c5242f None None] MFA Rules not processed for user `51033d56f85f4282a49ded6338440677`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:33:17.373988 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-4fddc781-cf24-43a9-8278-03f180c5242f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:17.466938 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1124/2234] 10.222.0.42 () {58 vars in 1018 bytes} [Thu Oct 27 01:33:17 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:33:17.479017 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd33b239-025c-421e-982a-6f798f575a5c None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:17.481113 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-bd33b239-025c-421e-982a-6f798f575a5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:17.549941 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd33b239-025c-421e-982a-6f798f575a5c None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:17.553566 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd93356-4c63-46d5-a829-29c1d3f16278 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2880dbb00eec471285d242e8183dd8d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:17.554238 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd93356-4c63-46d5-a829-29c1d3f16278 service glance] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:17.554581 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd93356-4c63-46d5-a829-29c1d3f16278 service glance] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:17.554910 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd93356-4c63-46d5-a829-29c1d3f16278 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:17.565425 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd93356-4c63-46d5-a829-29c1d3f16278 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:17.566904 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd93356-4c63-46d5-a829-29c1d3f16278 service glance] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:17.578073 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1111/2235] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:33:17 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:33:17.929313 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fddc781-cf24-43a9-8278-03f180c5242f None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:17.980832 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:33:17.997504 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fddc781-cf24-43a9-8278-03f180c5242f None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:18.002497 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8031d9c4-5dd5-4c96-b457-d5ebb764f188 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:18.003446 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8031d9c4-5dd5-4c96-b457-d5ebb764f188 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:18.003592 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8031d9c4-5dd5-4c96-b457-d5ebb764f188 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:18.003902 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8031d9c4-5dd5-4c96-b457-d5ebb764f188 admin admin] PATH_INFO: `/v3/users/583c0cdd199e47cdaa664b9e891472ae` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:18.005104 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8031d9c4-5dd5-4c96-b457-d5ebb764f188 admin admin] RBAC: Authorizing `identity:delete_user(user_id=583c0cdd199e47cdaa664b9e891472ae)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:18.007072 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8031d9c4-5dd5-4c96-b457-d5ebb764f188 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:18.054333 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-8031d9c4-5dd5-4c96-b457-d5ebb764f188 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '583c0cdd199e47cdaa664b9e891472ae'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:18.062756 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-8031d9c4-5dd5-4c96-b457-d5ebb764f188 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '583c0cdd199e47cdaa664b9e891472ae'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:18.076254 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-8031d9c4-5dd5-4c96-b457-d5ebb764f188 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '583c0cdd199e47cdaa664b9e891472ae'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:18.076946 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-8031d9c4-5dd5-4c96-b457-d5ebb764f188 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '583c0cdd199e47cdaa664b9e891472ae'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:18.077669 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-8031d9c4-5dd5-4c96-b457-d5ebb764f188 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '583c0cdd199e47cdaa664b9e891472ae'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:18.090460 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1125/2236] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:33:17 2022] DELETE /identity/v3/users/583c0cdd199e47cdaa664b9e891472ae => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:33:18.095135 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034017d7-9d80-4581-a73e-92a9a460e2a8 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:18.095942 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034017d7-9d80-4581-a73e-92a9a460e2a8 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:18.096221 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034017d7-9d80-4581-a73e-92a9a460e2a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:18.115637 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-034017d7-9d80-4581-a73e-92a9a460e2a8 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:33:18.116861 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-034017d7-9d80-4581-a73e-92a9a460e2a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:18.179081 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1112/2237] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:33:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:33:18.190115 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8031d9c4-5dd5-4c96-b457-d5ebb764f188 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:18.191649 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-8031d9c4-5dd5-4c96-b457-d5ebb764f188 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:18.255838 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8031d9c4-5dd5-4c96-b457-d5ebb764f188 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:18.259524 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc6e519-97b2-426d-be2f-2e227ebf8ac9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:18.260136 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc6e519-97b2-426d-be2f-2e227ebf8ac9 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:18.260409 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc6e519-97b2-426d-be2f-2e227ebf8ac9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:18.260678 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc6e519-97b2-426d-be2f-2e227ebf8ac9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:18.270790 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc6e519-97b2-426d-be2f-2e227ebf8ac9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:18.272250 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc6e519-97b2-426d-be2f-2e227ebf8ac9 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:18.283278 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1126/2238] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:33:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:33:18.590267 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034017d7-9d80-4581-a73e-92a9a460e2a8 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:18.592221 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-034017d7-9d80-4581-a73e-92a9a460e2a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:18.609005 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034017d7-9d80-4581-a73e-92a9a460e2a8 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:18.619316 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1f5317-0fe8-4ae3-8602-23e5a9e5efb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:18.620183 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1f5317-0fe8-4ae3-8602-23e5a9e5efb4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:18.620849 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1f5317-0fe8-4ae3-8602-23e5a9e5efb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:18.621169 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1f5317-0fe8-4ae3-8602-23e5a9e5efb4 admin admin] PATH_INFO: `/v3/projects/d64549dafdd9476fa6848f15c48e12e9` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:18.622348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c1f5317-0fe8-4ae3-8602-23e5a9e5efb4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d64549dafdd9476fa6848f15c48e12e9)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:18.624344 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c1f5317-0fe8-4ae3-8602-23e5a9e5efb4 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:18.679497 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1c1f5317-0fe8-4ae3-8602-23e5a9e5efb4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd64549dafdd9476fa6848f15c48e12e9', 'request_id': 'req-1c1f5317-0fe8-4ae3-8602-23e5a9e5efb4'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:18.688757 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1c1f5317-0fe8-4ae3-8602-23e5a9e5efb4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd64549dafdd9476fa6848f15c48e12e9', 'request_id': 'req-1c1f5317-0fe8-4ae3-8602-23e5a9e5efb4'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:18.695592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1c1f5317-0fe8-4ae3-8602-23e5a9e5efb4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd64549dafdd9476fa6848f15c48e12e9', 'request_id': 'req-1c1f5317-0fe8-4ae3-8602-23e5a9e5efb4'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:18.712467 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1c1f5317-0fe8-4ae3-8602-23e5a9e5efb4 admin admin] The token cache is being invalidate because project d64549dafdd9476fa6848f15c48e12e9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:33:18.712761 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1c1f5317-0fe8-4ae3-8602-23e5a9e5efb4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:18.713613 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1c1f5317-0fe8-4ae3-8602-23e5a9e5efb4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:18.715557 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1113/2239] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:33:18 2022] DELETE /identity/v3/projects/d64549dafdd9476fa6848f15c48e12e9 => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:33:18.719828 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc6e519-97b2-426d-be2f-2e227ebf8ac9 service neutron] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:18.731597 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc6e519-97b2-426d-be2f-2e227ebf8ac9 service neutron] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:18.789487 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4cc222-5443-46bb-92ec-1f0813137923 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:18.790106 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4cc222-5443-46bb-92ec-1f0813137923 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:18.790380 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4cc222-5443-46bb-92ec-1f0813137923 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:18.790649 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4cc222-5443-46bb-92ec-1f0813137923 admin admin] PATH_INFO: `/v3/users/e4ebdf87ba354977b89a50092027e10e` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:18.791750 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce4cc222-5443-46bb-92ec-1f0813137923 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e4ebdf87ba354977b89a50092027e10e)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:18.793698 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce4cc222-5443-46bb-92ec-1f0813137923 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:18.806357 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:33:18.841213 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-ce4cc222-5443-46bb-92ec-1f0813137923 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e4ebdf87ba354977b89a50092027e10e'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:18.850026 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-ce4cc222-5443-46bb-92ec-1f0813137923 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e4ebdf87ba354977b89a50092027e10e'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:18.862948 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-ce4cc222-5443-46bb-92ec-1f0813137923 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e4ebdf87ba354977b89a50092027e10e'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:18.863608 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-ce4cc222-5443-46bb-92ec-1f0813137923 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e4ebdf87ba354977b89a50092027e10e'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:18.864206 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-ce4cc222-5443-46bb-92ec-1f0813137923 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e4ebdf87ba354977b89a50092027e10e'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:18.877218 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1127/2240] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:33:18 2022] DELETE /identity/v3/users/e4ebdf87ba354977b89a50092027e10e => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:33:19.196344 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1f5317-0fe8-4ae3-8602-23e5a9e5efb4 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:19.197876 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-1c1f5317-0fe8-4ae3-8602-23e5a9e5efb4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:19.212772 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1f5317-0fe8-4ae3-8602-23e5a9e5efb4 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:19.217414 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:33:19.278325 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8341ed58-a2c4-40d2-823b-aeb863fc08ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:19.278951 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8341ed58-a2c4-40d2-823b-aeb863fc08ed admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:19.279229 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8341ed58-a2c4-40d2-823b-aeb863fc08ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:19.279498 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8341ed58-a2c4-40d2-823b-aeb863fc08ed admin admin] PATH_INFO: `/v3/projects/39f4415f672f45928fb4d2719f7c4ccd` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:19.280597 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8341ed58-a2c4-40d2-823b-aeb863fc08ed admin admin] RBAC: Authorizing `identity:delete_project(project_id=39f4415f672f45928fb4d2719f7c4ccd)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:19.282533 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8341ed58-a2c4-40d2-823b-aeb863fc08ed admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:19.328403 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-8341ed58-a2c4-40d2-823b-aeb863fc08ed admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '39f4415f672f45928fb4d2719f7c4ccd', 'request_id': 'req-8341ed58-a2c4-40d2-823b-aeb863fc08ed'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:19.336484 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-8341ed58-a2c4-40d2-823b-aeb863fc08ed admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '39f4415f672f45928fb4d2719f7c4ccd', 'request_id': 'req-8341ed58-a2c4-40d2-823b-aeb863fc08ed'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:19.342423 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-8341ed58-a2c4-40d2-823b-aeb863fc08ed admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '39f4415f672f45928fb4d2719f7c4ccd', 'request_id': 'req-8341ed58-a2c4-40d2-823b-aeb863fc08ed'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:19.353923 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-8341ed58-a2c4-40d2-823b-aeb863fc08ed admin admin] The token cache is being invalidate because project 39f4415f672f45928fb4d2719f7c4ccd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:33:19.354209 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-8341ed58-a2c4-40d2-823b-aeb863fc08ed admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:19.354836 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-8341ed58-a2c4-40d2-823b-aeb863fc08ed admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:19.356298 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1114/2241] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:33:19 2022] DELETE /identity/v3/projects/39f4415f672f45928fb4d2719f7c4ccd => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:33:19.362036 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858328a2-9dd1-453c-a447-1d7437d67d64 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:19.362263 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858328a2-9dd1-453c-a447-1d7437d67d64 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:19.362535 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858328a2-9dd1-453c-a447-1d7437d67d64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:19.382199 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-858328a2-9dd1-453c-a447-1d7437d67d64 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:33:19.383377 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-858328a2-9dd1-453c-a447-1d7437d67d64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:19.442779 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1128/2242] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:33:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:33:19.450599 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296efa7f-49cd-4b3c-885c-0580463806df None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:19.450884 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296efa7f-49cd-4b3c-885c-0580463806df None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:19.451160 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296efa7f-49cd-4b3c-885c-0580463806df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:19.471267 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-296efa7f-49cd-4b3c-885c-0580463806df None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:33:19.472462 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-296efa7f-49cd-4b3c-885c-0580463806df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:19.480248 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1115/2243] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:33:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:33:19.484991 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858328a2-9dd1-453c-a447-1d7437d67d64 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:19.494778 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858328a2-9dd1-453c-a447-1d7437d67d64 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:19.499263 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd936c06-8b4f-418d-a792-8e1f4e4aaf88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:19.499851 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd936c06-8b4f-418d-a792-8e1f4e4aaf88 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:19.500126 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd936c06-8b4f-418d-a792-8e1f4e4aaf88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:19.500400 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd936c06-8b4f-418d-a792-8e1f4e4aaf88 admin admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:19.501040 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd936c06-8b4f-418d-a792-8e1f4e4aaf88 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:33:19.501367 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd936c06-8b4f-418d-a792-8e1f4e4aaf88 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:19.503672 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd936c06-8b4f-418d-a792-8e1f4e4aaf88 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:19.514967 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1129/2244] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:33:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:33:19.519946 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296efa7f-49cd-4b3c-885c-0580463806df None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:19.529035 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296efa7f-49cd-4b3c-885c-0580463806df None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:19.534180 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4afcc3b-47ca-4caa-86a6-e83671fb71ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:19.534773 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4afcc3b-47ca-4caa-86a6-e83671fb71ac admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:19.535051 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4afcc3b-47ca-4caa-86a6-e83671fb71ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:19.535323 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4afcc3b-47ca-4caa-86a6-e83671fb71ac admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:19.535977 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4afcc3b-47ca-4caa-86a6-e83671fb71ac admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:19.537915 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4afcc3b-47ca-4caa-86a6-e83671fb71ac admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:19.563931 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1116/2245] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:33:19 2022] POST /identity/v3/projects => generated 367 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:33:19.568646 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd936c06-8b4f-418d-a792-8e1f4e4aaf88 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:19.578643 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd936c06-8b4f-418d-a792-8e1f4e4aaf88 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:19.635485 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0182a11-4e77-4d8d-bb12-bb56bca471eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:19.636015 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0182a11-4e77-4d8d-bb12-bb56bca471eb admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:19.636291 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0182a11-4e77-4d8d-bb12-bb56bca471eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:19.636562 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0182a11-4e77-4d8d-bb12-bb56bca471eb admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:19.637243 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0182a11-4e77-4d8d-bb12-bb56bca471eb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:19.639148 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0182a11-4e77-4d8d-bb12-bb56bca471eb admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:19.660162 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1130/2246] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:33:19 2022] POST /identity/v3/users => generated 284 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:33:19.664361 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4afcc3b-47ca-4caa-86a6-e83671fb71ac admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:19.675324 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4afcc3b-47ca-4caa-86a6-e83671fb71ac admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:19.679922 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbfeeb7e-9d72-42d5-91d3-3d5b46372608 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:19.680510 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfeeb7e-9d72-42d5-91d3-3d5b46372608 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:19.680788 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfeeb7e-9d72-42d5-91d3-3d5b46372608 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:19.681070 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfeeb7e-9d72-42d5-91d3-3d5b46372608 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:19.681723 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbfeeb7e-9d72-42d5-91d3-3d5b46372608 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:19.683310 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbfeeb7e-9d72-42d5-91d3-3d5b46372608 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:19.693761 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1117/2247] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:33:19 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:33:19.697656 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0182a11-4e77-4d8d-bb12-bb56bca471eb admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:19.707983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0182a11-4e77-4d8d-bb12-bb56bca471eb admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:19.712946 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c119786-810e-4736-b9f9-288e9fdcd2f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:19.713576 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c119786-810e-4736-b9f9-288e9fdcd2f1 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:19.713859 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c119786-810e-4736-b9f9-288e9fdcd2f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:19.714129 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c119786-810e-4736-b9f9-288e9fdcd2f1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:19.714802 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c119786-810e-4736-b9f9-288e9fdcd2f1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:19.716390 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c119786-810e-4736-b9f9-288e9fdcd2f1 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:19.717949 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=87,keystone:DELETE=9,keystone:INSERT=4 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:33:19.730033 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1131/2248] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:33:19 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:33:19.733885 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbfeeb7e-9d72-42d5-91d3-3d5b46372608 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:19.744608 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbfeeb7e-9d72-42d5-91d3-3d5b46372608 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:19.749319 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92174089-ac12-47a4-a659-e7c5e2609268 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:19.749970 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92174089-ac12-47a4-a659-e7c5e2609268 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:19.750249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92174089-ac12-47a4-a659-e7c5e2609268 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:19.750520 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92174089-ac12-47a4-a659-e7c5e2609268 admin admin] PATH_INFO: `/v3/projects/5c697d48969f44589359cbfb5952b34f/users/a013674911804dff8ed3c5b186d9abc7/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:19.766444 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92174089-ac12-47a4-a659-e7c5e2609268 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5c697d48969f44589359cbfb5952b34f, user_id=a013674911804dff8ed3c5b186d9abc7, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:19.769347 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92174089-ac12-47a4-a659-e7c5e2609268 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:19.783572 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1118/2249] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:33:19 2022] PUT /identity/v3/projects/5c697d48969f44589359cbfb5952b34f/users/a013674911804dff8ed3c5b186d9abc7/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:33:19.793969 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c041878-e5f1-4bd3-8408-49ecdc2a0da7 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:19.794252 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c041878-e5f1-4bd3-8408-49ecdc2a0da7 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:19.794569 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c041878-e5f1-4bd3-8408-49ecdc2a0da7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:19.824810 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-8c041878-e5f1-4bd3-8408-49ecdc2a0da7 None None] MFA Rules not processed for user `a013674911804dff8ed3c5b186d9abc7`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:33:19.826220 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-8c041878-e5f1-4bd3-8408-49ecdc2a0da7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:19.891961 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1132/2250] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:33:19 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:33:19.904425 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92174089-ac12-47a4-a659-e7c5e2609268 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:19.906191 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-92174089-ac12-47a4-a659-e7c5e2609268 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:19.945976 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:33:19.964419 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92174089-ac12-47a4-a659-e7c5e2609268 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:19.968013 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-372aa4af-abc5-444b-8ff3-41eef46684b3 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2880dbb00eec471285d242e8183dd8d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:19.968610 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372aa4af-abc5-444b-8ff3-41eef46684b3 service glance] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:19.968887 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372aa4af-abc5-444b-8ff3-41eef46684b3 service glance] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:19.969196 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372aa4af-abc5-444b-8ff3-41eef46684b3 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:19.977882 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-372aa4af-abc5-444b-8ff3-41eef46684b3 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:19.979311 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-372aa4af-abc5-444b-8ff3-41eef46684b3 service glance] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:19.988119 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1119/2251] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:33:19 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:33:22.271613 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f79ce7-8a20-44cb-9238-11d25b4be7fa None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:22.272123 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f79ce7-8a20-44cb-9238-11d25b4be7fa None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:22.272207 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f79ce7-8a20-44cb-9238-11d25b4be7fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:22.309870 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-c9f79ce7-8a20-44cb-9238-11d25b4be7fa None None] MFA Rules not processed for user `7a3f8b5eb09f4ba29f4bd343bd91f6a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:33:22.311141 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-c9f79ce7-8a20-44cb-9238-11d25b4be7fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:22.380453 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1133/2252] 10.222.0.42 () {58 vars in 1009 bytes} [Thu Oct 27 01:33:22 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:33:22.394057 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-372aa4af-abc5-444b-8ff3-41eef46684b3 service glance] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:22.395492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-372aa4af-abc5-444b-8ff3-41eef46684b3 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:22.480497 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-372aa4af-abc5-444b-8ff3-41eef46684b3 service glance] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:22.487327 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77da2359-c376-4ce8-8d63-7b68b7a933dd service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77f2cf9e0c3541f9bccfb7e213d88609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:22.488022 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77da2359-c376-4ce8-8d63-7b68b7a933dd service swift] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:22.488293 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77da2359-c376-4ce8-8d63-7b68b7a933dd service swift] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:22.488603 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77da2359-c376-4ce8-8d63-7b68b7a933dd service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:22.497501 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77da2359-c376-4ce8-8d63-7b68b7a933dd service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:22.499018 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77da2359-c376-4ce8-8d63-7b68b7a933dd service swift] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:22.507703 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1120/2253] 10.222.0.42 () {60 vars in 1384 bytes} [Thu Oct 27 01:33:22 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 27 01:33:32.373789 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=59 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:33:32.504542 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=42 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:33:47.403196 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f79ce7-8a20-44cb-9238-11d25b4be7fa None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:47.404580 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-c9f79ce7-8a20-44cb-9238-11d25b4be7fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:47.460033 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f79ce7-8a20-44cb-9238-11d25b4be7fa None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:47.463562 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96b7c3c-4064-4705-b4af-a14684f52d21 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:47.464150 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96b7c3c-4064-4705-b4af-a14684f52d21 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:47.464429 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96b7c3c-4064-4705-b4af-a14684f52d21 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:47.464700 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96b7c3c-4064-4705-b4af-a14684f52d21 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:47.466386 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-c96b7c3c-4064-4705-b4af-a14684f52d21 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:47.527944 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c96b7c3c-4064-4705-b4af-a14684f52d21 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:47.529399 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c96b7c3c-4064-4705-b4af-a14684f52d21 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:47.539677 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1134/2254] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:33:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:33:51.085109 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77da2359-c376-4ce8-8d63-7b68b7a933dd service swift] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:51.086688 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-77da2359-c376-4ce8-8d63-7b68b7a933dd service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:51.102175 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77da2359-c376-4ce8-8d63-7b68b7a933dd service swift] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:51.106836 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e239a7-8925-4ab5-bfe2-41c72fd4dc20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:51.107474 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e239a7-8925-4ab5-bfe2-41c72fd4dc20 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:51.107909 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e239a7-8925-4ab5-bfe2-41c72fd4dc20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:51.108284 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e239a7-8925-4ab5-bfe2-41c72fd4dc20 admin admin] PATH_INFO: `/v3/users/99bcd22ff73b4526a54cf0e76f509436` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:51.125497 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e239a7-8925-4ab5-bfe2-41c72fd4dc20 admin admin] RBAC: Authorizing `identity:delete_user(user_id=99bcd22ff73b4526a54cf0e76f509436)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:51.128005 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e239a7-8925-4ab5-bfe2-41c72fd4dc20 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:51.173739 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-96e239a7-8925-4ab5-bfe2-41c72fd4dc20 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '99bcd22ff73b4526a54cf0e76f509436'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:51.181362 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-96e239a7-8925-4ab5-bfe2-41c72fd4dc20 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '99bcd22ff73b4526a54cf0e76f509436'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:51.181978 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-96e239a7-8925-4ab5-bfe2-41c72fd4dc20 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '99bcd22ff73b4526a54cf0e76f509436'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:51.195165 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-96e239a7-8925-4ab5-bfe2-41c72fd4dc20 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '99bcd22ff73b4526a54cf0e76f509436'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:51.202191 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-96e239a7-8925-4ab5-bfe2-41c72fd4dc20 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '99bcd22ff73b4526a54cf0e76f509436'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:51.208799 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1121/2255] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:33:51 2022] DELETE /identity/v3/users/99bcd22ff73b4526a54cf0e76f509436 => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:33:51.381869 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96b7c3c-4064-4705-b4af-a14684f52d21 service neutron] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:51.394505 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96b7c3c-4064-4705-b4af-a14684f52d21 service neutron] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:51.451472 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65f1e161-0caa-437a-971c-4085daa948c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:51.452096 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f1e161-0caa-437a-971c-4085daa948c6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:51.452373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f1e161-0caa-437a-971c-4085daa948c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:51.452644 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f1e161-0caa-437a-971c-4085daa948c6 admin admin] PATH_INFO: `/v3/projects/db70dacc270d431182451b57b277484f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:51.463489 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65f1e161-0caa-437a-971c-4085daa948c6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=db70dacc270d431182451b57b277484f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:51.465439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65f1e161-0caa-437a-971c-4085daa948c6 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:51.507930 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:33:51.511913 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-65f1e161-0caa-437a-971c-4085daa948c6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'db70dacc270d431182451b57b277484f', 'request_id': 'req-65f1e161-0caa-437a-971c-4085daa948c6'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:51.520823 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-65f1e161-0caa-437a-971c-4085daa948c6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'db70dacc270d431182451b57b277484f', 'request_id': 'req-65f1e161-0caa-437a-971c-4085daa948c6'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:51.528421 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-65f1e161-0caa-437a-971c-4085daa948c6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'db70dacc270d431182451b57b277484f', 'request_id': 'req-65f1e161-0caa-437a-971c-4085daa948c6'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:51.540214 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-65f1e161-0caa-437a-971c-4085daa948c6 admin admin] The token cache is being invalidate because project db70dacc270d431182451b57b277484f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:33:51.540531 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-65f1e161-0caa-437a-971c-4085daa948c6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:51.541263 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-65f1e161-0caa-437a-971c-4085daa948c6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:33:51.542753 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1135/2256] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:33:51 2022] DELETE /identity/v3/projects/db70dacc270d431182451b57b277484f => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:33:51.548684 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75c9cba-34ef-41ec-b160-6d8a08568278 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:51.549741 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75c9cba-34ef-41ec-b160-6d8a08568278 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:51.550010 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75c9cba-34ef-41ec-b160-6d8a08568278 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:51.570939 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-e75c9cba-34ef-41ec-b160-6d8a08568278 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:33:51.572142 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e75c9cba-34ef-41ec-b160-6d8a08568278 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:51.639447 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1122/2257] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:33:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:33:51.648172 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04cfb29d-d800-452d-9c1b-3e8377479286 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:51.648504 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04cfb29d-d800-452d-9c1b-3e8377479286 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:51.648780 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04cfb29d-d800-452d-9c1b-3e8377479286 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:51.668823 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-04cfb29d-d800-452d-9c1b-3e8377479286 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:33:51.670046 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-04cfb29d-d800-452d-9c1b-3e8377479286 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:51.680546 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1136/2258] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:33:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:33:51.684650 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e75c9cba-34ef-41ec-b160-6d8a08568278 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:51.695322 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e75c9cba-34ef-41ec-b160-6d8a08568278 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:51.699906 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8887555-a2b3-427b-acd2-4381c6df08a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:51.700503 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8887555-a2b3-427b-acd2-4381c6df08a1 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:51.700780 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8887555-a2b3-427b-acd2-4381c6df08a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:51.701052 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8887555-a2b3-427b-acd2-4381c6df08a1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:51.701732 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8887555-a2b3-427b-acd2-4381c6df08a1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:33:51.702036 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8887555-a2b3-427b-acd2-4381c6df08a1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:51.703634 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8887555-a2b3-427b-acd2-4381c6df08a1 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:51.722768 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1123/2259] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:33:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:33:51.723416 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04cfb29d-d800-452d-9c1b-3e8377479286 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:51.733230 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04cfb29d-d800-452d-9c1b-3e8377479286 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:51.738541 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc704a40-1f9a-4b3e-b550-c64ef8fc3560 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:51.739299 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc704a40-1f9a-4b3e-b550-c64ef8fc3560 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:51.739774 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc704a40-1f9a-4b3e-b550-c64ef8fc3560 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:51.740003 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc704a40-1f9a-4b3e-b550-c64ef8fc3560 admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:51.740958 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc704a40-1f9a-4b3e-b550-c64ef8fc3560 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:51.743214 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc704a40-1f9a-4b3e-b550-c64ef8fc3560 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:51.787536 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1137/2260] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:33:51 2022] POST /identity/v3/projects => generated 385 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:33:51.791930 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8887555-a2b3-427b-acd2-4381c6df08a1 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:51.810439 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8887555-a2b3-427b-acd2-4381c6df08a1 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:51.882531 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:33:51.895175 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986dabc9-f526-4e56-a09c-dfcc4a0e3e77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:51.895785 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986dabc9-f526-4e56-a09c-dfcc4a0e3e77 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:51.896067 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986dabc9-f526-4e56-a09c-dfcc4a0e3e77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:51.896312 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986dabc9-f526-4e56-a09c-dfcc4a0e3e77 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:51.896938 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-986dabc9-f526-4e56-a09c-dfcc4a0e3e77 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:51.898894 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-986dabc9-f526-4e56-a09c-dfcc4a0e3e77 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:51.926685 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1124/2261] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:33:51 2022] POST /identity/v3/users => generated 293 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:33:51.935529 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc704a40-1f9a-4b3e-b550-c64ef8fc3560 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:51.950049 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc704a40-1f9a-4b3e-b550-c64ef8fc3560 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:51.957023 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99df5129-1881-484b-9eed-e086c4ff9ac0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:51.957682 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99df5129-1881-484b-9eed-e086c4ff9ac0 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:51.957987 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99df5129-1881-484b-9eed-e086c4ff9ac0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:51.958329 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99df5129-1881-484b-9eed-e086c4ff9ac0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:51.959027 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99df5129-1881-484b-9eed-e086c4ff9ac0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:51.960656 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99df5129-1881-484b-9eed-e086c4ff9ac0 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:51.972541 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1138/2262] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:33:51 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:33:51.976794 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986dabc9-f526-4e56-a09c-dfcc4a0e3e77 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:51.988129 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986dabc9-f526-4e56-a09c-dfcc4a0e3e77 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:51.992681 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785d80b4-12d0-4d67-ab85-e00df31555fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:51.993307 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785d80b4-12d0-4d67-ab85-e00df31555fd admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:51.993582 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785d80b4-12d0-4d67-ab85-e00df31555fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:51.993850 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785d80b4-12d0-4d67-ab85-e00df31555fd admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:51.994487 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-785d80b4-12d0-4d67-ab85-e00df31555fd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:51.996063 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-785d80b4-12d0-4d67-ab85-e00df31555fd admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:52.008092 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1125/2263] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:33:51 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:33:52.012413 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99df5129-1881-484b-9eed-e086c4ff9ac0 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:52.023786 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99df5129-1881-484b-9eed-e086c4ff9ac0 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:52.028417 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a27ac93-756c-4b11-8eb4-77a437c64a71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:52.029099 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a27ac93-756c-4b11-8eb4-77a437c64a71 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:52.029362 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a27ac93-756c-4b11-8eb4-77a437c64a71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:52.029636 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a27ac93-756c-4b11-8eb4-77a437c64a71 admin admin] PATH_INFO: `/v3/projects/0e3cfa878c494ae18fd619116fa25466/users/07896f1bd26c47d8bfed3b0ad7b7e90c/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:52.044813 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a27ac93-756c-4b11-8eb4-77a437c64a71 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0e3cfa878c494ae18fd619116fa25466, user_id=07896f1bd26c47d8bfed3b0ad7b7e90c, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:52.047768 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a27ac93-756c-4b11-8eb4-77a437c64a71 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:52.061631 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1139/2264] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:33:52 2022] PUT /identity/v3/projects/0e3cfa878c494ae18fd619116fa25466/users/07896f1bd26c47d8bfed3b0ad7b7e90c/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:33:52.073974 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bcc3f5-683a-44d8-85b9-b3e7c8155ffe None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:52.074261 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bcc3f5-683a-44d8-85b9-b3e7c8155ffe None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:52.074571 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bcc3f5-683a-44d8-85b9-b3e7c8155ffe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:52.103990 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-a0bcc3f5-683a-44d8-85b9-b3e7c8155ffe None None] MFA Rules not processed for user `07896f1bd26c47d8bfed3b0ad7b7e90c`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:33:52.105223 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-a0bcc3f5-683a-44d8-85b9-b3e7c8155ffe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:52.168994 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1126/2265] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:33:52 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:33:52.173607 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a27ac93-756c-4b11-8eb4-77a437c64a71 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:52.237636 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a27ac93-756c-4b11-8eb4-77a437c64a71 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:52.242331 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfb5628-a050-4bb2-a571-00db0c2e3cbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:52.242933 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfb5628-a050-4bb2-a571-00db0c2e3cbe admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:52.243210 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfb5628-a050-4bb2-a571-00db0c2e3cbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:52.243476 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfb5628-a050-4bb2-a571-00db0c2e3cbe admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:52.244105 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bfb5628-a050-4bb2-a571-00db0c2e3cbe admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:52.246172 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bfb5628-a050-4bb2-a571-00db0c2e3cbe admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:52.274537 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1140/2266] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:33:52 2022] POST /identity/v3/projects => generated 387 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:33:52.278526 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0bcc3f5-683a-44d8-85b9-b3e7c8155ffe None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:52.290115 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0bcc3f5-683a-44d8-85b9-b3e7c8155ffe None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:52.351929 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c197f94f-8e47-4def-a4a8-93975220ce8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:52.352627 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c197f94f-8e47-4def-a4a8-93975220ce8f admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:52.353007 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c197f94f-8e47-4def-a4a8-93975220ce8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:52.353690 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c197f94f-8e47-4def-a4a8-93975220ce8f admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:52.354154 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c197f94f-8e47-4def-a4a8-93975220ce8f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:52.356140 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c197f94f-8e47-4def-a4a8-93975220ce8f admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:52.382090 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1127/2267] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:33:52 2022] POST /identity/v3/users => generated 294 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:33:52.383374 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfb5628-a050-4bb2-a571-00db0c2e3cbe admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:52.394083 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfb5628-a050-4bb2-a571-00db0c2e3cbe admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:52.399070 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5d1198-bb70-4b7c-ad54-c69dc9661c39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:52.399755 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5d1198-bb70-4b7c-ad54-c69dc9661c39 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:52.400105 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5d1198-bb70-4b7c-ad54-c69dc9661c39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:52.400449 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5d1198-bb70-4b7c-ad54-c69dc9661c39 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:52.401203 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5d1198-bb70-4b7c-ad54-c69dc9661c39 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:52.402879 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5d1198-bb70-4b7c-ad54-c69dc9661c39 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:52.415016 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1141/2268] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:33:52 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:33:52.419261 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c197f94f-8e47-4def-a4a8-93975220ce8f admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:52.430990 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c197f94f-8e47-4def-a4a8-93975220ce8f admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:52.435636 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e133d0-5da6-43da-8db3-333798f031fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:52.436251 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e133d0-5da6-43da-8db3-333798f031fa admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:52.436533 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e133d0-5da6-43da-8db3-333798f031fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:52.436804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e133d0-5da6-43da-8db3-333798f031fa admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:52.437505 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67e133d0-5da6-43da-8db3-333798f031fa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:52.439104 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67e133d0-5da6-43da-8db3-333798f031fa admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:52.449136 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1128/2269] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:33:52 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:33:52.453382 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5d1198-bb70-4b7c-ad54-c69dc9661c39 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:52.463563 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5d1198-bb70-4b7c-ad54-c69dc9661c39 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:52.468192 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ad9c9f-2608-40fc-aa1b-65530e136d4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:52.468839 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ad9c9f-2608-40fc-aa1b-65530e136d4b admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:52.469149 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ad9c9f-2608-40fc-aa1b-65530e136d4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:52.469442 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ad9c9f-2608-40fc-aa1b-65530e136d4b admin admin] PATH_INFO: `/v3/projects/919490995bfa4c2b8084cfbcdb2dcb01/users/fd38cadd45944b428959df77c29148ea/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:52.484957 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ad9c9f-2608-40fc-aa1b-65530e136d4b admin admin] RBAC: Authorizing `identity:create_grant(project_id=919490995bfa4c2b8084cfbcdb2dcb01, user_id=fd38cadd45944b428959df77c29148ea, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:52.488712 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ad9c9f-2608-40fc-aa1b-65530e136d4b admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:52.501488 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1142/2270] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:33:52 2022] PUT /identity/v3/projects/919490995bfa4c2b8084cfbcdb2dcb01/users/fd38cadd45944b428959df77c29148ea/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:33:52.512372 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c6b08c-98ec-4402-8f15-e1b4420b587d None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:52.512658 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c6b08c-98ec-4402-8f15-e1b4420b587d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:52.512934 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c6b08c-98ec-4402-8f15-e1b4420b587d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:52.541195 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-41c6b08c-98ec-4402-8f15-e1b4420b587d None None] MFA Rules not processed for user `fd38cadd45944b428959df77c29148ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:33:52.542589 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-41c6b08c-98ec-4402-8f15-e1b4420b587d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:52.567006 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:33:52.607514 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1129/2271] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:33:52 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:33:52.612679 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ad9c9f-2608-40fc-aa1b-65530e136d4b admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:52.623069 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=91,keystone:DELETE=4,keystone:INSERT=5 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:33:52.678297 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ad9c9f-2608-40fc-aa1b-65530e136d4b admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:52.682996 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de6be2dd-9ea3-4e51-adbb-7231ce4fa7c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:52.683598 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6be2dd-9ea3-4e51-adbb-7231ce4fa7c2 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:52.683879 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6be2dd-9ea3-4e51-adbb-7231ce4fa7c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:52.684154 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6be2dd-9ea3-4e51-adbb-7231ce4fa7c2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:52.684793 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de6be2dd-9ea3-4e51-adbb-7231ce4fa7c2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:52.686795 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de6be2dd-9ea3-4e51-adbb-7231ce4fa7c2 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:52.715506 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1143/2272] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:33:52 2022] POST /identity/v3/projects => generated 387 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:33:52.719705 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c6b08c-98ec-4402-8f15-e1b4420b587d None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:52.731619 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c6b08c-98ec-4402-8f15-e1b4420b587d None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:52.792924 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9abb89f1-62fd-4fe3-ad31-3c62fadf2124 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:52.793566 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abb89f1-62fd-4fe3-ad31-3c62fadf2124 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:52.793853 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abb89f1-62fd-4fe3-ad31-3c62fadf2124 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:52.794130 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abb89f1-62fd-4fe3-ad31-3c62fadf2124 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:52.794769 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9abb89f1-62fd-4fe3-ad31-3c62fadf2124 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:52.796692 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9abb89f1-62fd-4fe3-ad31-3c62fadf2124 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:52.820805 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1130/2273] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:33:52 2022] POST /identity/v3/users => generated 300 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:33:52.824664 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de6be2dd-9ea3-4e51-adbb-7231ce4fa7c2 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:52.835892 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de6be2dd-9ea3-4e51-adbb-7231ce4fa7c2 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:52.840650 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e55e38-7097-4b6c-ba09-be517ca668a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:52.841294 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e55e38-7097-4b6c-ba09-be517ca668a1 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:52.841570 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e55e38-7097-4b6c-ba09-be517ca668a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:52.841837 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e55e38-7097-4b6c-ba09-be517ca668a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:52.842508 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e55e38-7097-4b6c-ba09-be517ca668a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:52.844131 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e55e38-7097-4b6c-ba09-be517ca668a1 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:52.855653 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1144/2274] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:33:52 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:33:52.859959 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9abb89f1-62fd-4fe3-ad31-3c62fadf2124 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:52.870147 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9abb89f1-62fd-4fe3-ad31-3c62fadf2124 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:52.874762 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f86ed1-8ec2-4137-b257-d2193420c327 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:52.875411 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f86ed1-8ec2-4137-b257-d2193420c327 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:52.875694 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f86ed1-8ec2-4137-b257-d2193420c327 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:52.875963 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f86ed1-8ec2-4137-b257-d2193420c327 admin admin] PATH_INFO: `/v3/projects/1767f1f9014b4a6e99e343757adf03cf/users/c031fd72c2104217bf4b9a7587ad438b/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:52.892397 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f86ed1-8ec2-4137-b257-d2193420c327 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1767f1f9014b4a6e99e343757adf03cf, user_id=c031fd72c2104217bf4b9a7587ad438b, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:52.895941 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f86ed1-8ec2-4137-b257-d2193420c327 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:52.912565 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1131/2275] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:33:52 2022] PUT /identity/v3/projects/1767f1f9014b4a6e99e343757adf03cf/users/c031fd72c2104217bf4b9a7587ad438b/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:33:52.916672 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e55e38-7097-4b6c-ba09-be517ca668a1 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:52.928069 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e55e38-7097-4b6c-ba09-be517ca668a1 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:52.989703 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d7759f-05d6-4705-bcaa-60892569962c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:52.990407 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d7759f-05d6-4705-bcaa-60892569962c admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:52.990709 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d7759f-05d6-4705-bcaa-60892569962c admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:52.991210 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d7759f-05d6-4705-bcaa-60892569962c admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:52.991802 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d7759f-05d6-4705-bcaa-60892569962c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:52.993446 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d7759f-05d6-4705-bcaa-60892569962c admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:53.004706 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1145/2276] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:33:52 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 27 01:33:53.009444 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f86ed1-8ec2-4137-b257-d2193420c327 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:53.020583 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f86ed1-8ec2-4137-b257-d2193420c327 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:53.026384 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94507edb-869a-4637-a4b5-b0d21f3ca0b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:53.027176 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94507edb-869a-4637-a4b5-b0d21f3ca0b1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:53.027683 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94507edb-869a-4637-a4b5-b0d21f3ca0b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:53.028211 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94507edb-869a-4637-a4b5-b0d21f3ca0b1 admin admin] PATH_INFO: `/v3/projects/1767f1f9014b4a6e99e343757adf03cf/users/c031fd72c2104217bf4b9a7587ad438b/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:53.034347 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94507edb-869a-4637-a4b5-b0d21f3ca0b1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1767f1f9014b4a6e99e343757adf03cf, user_id=c031fd72c2104217bf4b9a7587ad438b, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:53.037338 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94507edb-869a-4637-a4b5-b0d21f3ca0b1 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:53.049728 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1132/2277] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:33:53 2022] PUT /identity/v3/projects/1767f1f9014b4a6e99e343757adf03cf/users/c031fd72c2104217bf4b9a7587ad438b/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:33:53.060043 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db180ff0-6609-461d-9f55-a1aa0c216bdf None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:53.060338 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db180ff0-6609-461d-9f55-a1aa0c216bdf None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:53.060619 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db180ff0-6609-461d-9f55-a1aa0c216bdf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:53.091086 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-db180ff0-6609-461d-9f55-a1aa0c216bdf None None] MFA Rules not processed for user `c031fd72c2104217bf4b9a7587ad438b`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:33:53.092415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-db180ff0-6609-461d-9f55-a1aa0c216bdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:53.165082 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1146/2278] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:33:53 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:33:53.177183 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94507edb-869a-4637-a4b5-b0d21f3ca0b1 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:53.179029 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-94507edb-869a-4637-a4b5-b0d21f3ca0b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:53.241074 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94507edb-869a-4637-a4b5-b0d21f3ca0b1 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:53.244787 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13be7403-5df4-40ea-a379-fa190edd97db service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:53.245581 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13be7403-5df4-40ea-a379-fa190edd97db service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:53.245786 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13be7403-5df4-40ea-a379-fa190edd97db service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:53.246075 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13be7403-5df4-40ea-a379-fa190edd97db service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:53.256064 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13be7403-5df4-40ea-a379-fa190edd97db service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:53.257590 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13be7403-5df4-40ea-a379-fa190edd97db service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:53.273378 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1133/2279] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:33:53 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 27 01:33:53.779988 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db180ff0-6609-461d-9f55-a1aa0c216bdf None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:53.785680 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:33:53.789998 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db180ff0-6609-461d-9f55-a1aa0c216bdf None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:53.794508 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66da96cb-0f9a-428c-98eb-6001c97f9f8e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:53.795315 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66da96cb-0f9a-428c-98eb-6001c97f9f8e service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:53.795604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66da96cb-0f9a-428c-98eb-6001c97f9f8e service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:53.795885 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66da96cb-0f9a-428c-98eb-6001c97f9f8e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:53.852873 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66da96cb-0f9a-428c-98eb-6001c97f9f8e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:53.854334 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66da96cb-0f9a-428c-98eb-6001c97f9f8e service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:53.863051 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1147/2280] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:33:53 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:33:57.910691 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29391fd-656e-4e41-9d24-da2d88ad7962 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:57.911196 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29391fd-656e-4e41-9d24-da2d88ad7962 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:57.911325 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29391fd-656e-4e41-9d24-da2d88ad7962 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:57.930825 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-e29391fd-656e-4e41-9d24-da2d88ad7962 None None] MFA Rules not processed for user `7a3f8b5eb09f4ba29f4bd343bd91f6a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:33:57.932140 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e29391fd-656e-4e41-9d24-da2d88ad7962 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:57.948068 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:33:57.990036 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1134/2281] 10.222.0.42 () {58 vars in 967 bytes} [Thu Oct 27 01:33:57 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:33:58.003223 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66da96cb-0f9a-428c-98eb-6001c97f9f8e service cinder] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:33:58.004683 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-66da96cb-0f9a-428c-98eb-6001c97f9f8e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:33:58.072924 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66da96cb-0f9a-428c-98eb-6001c97f9f8e service cinder] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:33:58.077891 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e85f6f7-cc56-4cdd-9288-4359c1e9953e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77f2cf9e0c3541f9bccfb7e213d88609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:33:58.078493 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e85f6f7-cc56-4cdd-9288-4359c1e9953e service swift] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:33:58.078772 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e85f6f7-cc56-4cdd-9288-4359c1e9953e service swift] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:33:58.079043 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e85f6f7-cc56-4cdd-9288-4359c1e9953e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:33:58.087245 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e85f6f7-cc56-4cdd-9288-4359c1e9953e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:33:58.088828 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e85f6f7-cc56-4cdd-9288-4359c1e9953e service swift] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:33:58.096767 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1148/2282] 10.222.0.42 () {60 vars in 1384 bytes} [Thu Oct 27 01:33:58 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 27 01:34:02.102797 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e29391fd-656e-4e41-9d24-da2d88ad7962 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:02.104208 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e29391fd-656e-4e41-9d24-da2d88ad7962 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:02.115555 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e29391fd-656e-4e41-9d24-da2d88ad7962 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:02.119075 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff5f325-06ba-4a70-8a58-2a6786364ed9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:02.119669 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff5f325-06ba-4a70-8a58-2a6786364ed9 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:02.119950 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff5f325-06ba-4a70-8a58-2a6786364ed9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:02.120219 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff5f325-06ba-4a70-8a58-2a6786364ed9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:02.177474 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff5f325-06ba-4a70-8a58-2a6786364ed9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:02.178890 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff5f325-06ba-4a70-8a58-2a6786364ed9 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:02.187849 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1135/2283] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:34:02 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:34:04.261233 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e85f6f7-cc56-4cdd-9288-4359c1e9953e service swift] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:04.263033 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-8e85f6f7-cc56-4cdd-9288-4359c1e9953e service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:04.267219 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff5f325-06ba-4a70-8a58-2a6786364ed9 service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:04.332002 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff5f325-06ba-4a70-8a58-2a6786364ed9 service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:04.332952 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e85f6f7-cc56-4cdd-9288-4359c1e9953e service swift] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:04.336423 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb2e5cc-b308-4af3-8880-32920ec741f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:04.337079 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb2e5cc-b308-4af3-8880-32920ec741f3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:04.337342 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb2e5cc-b308-4af3-8880-32920ec741f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:04.337613 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb2e5cc-b308-4af3-8880-32920ec741f3 admin admin] PATH_INFO: `/v3/users/07896f1bd26c47d8bfed3b0ad7b7e90c` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:04.337982 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-682e9a88-a9f8-40b9-811c-8c7808c1dab6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:04.338613 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682e9a88-a9f8-40b9-811c-8c7808c1dab6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:04.338740 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb2e5cc-b308-4af3-8880-32920ec741f3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=07896f1bd26c47d8bfed3b0ad7b7e90c)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:04.338940 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682e9a88-a9f8-40b9-811c-8c7808c1dab6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:04.339161 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682e9a88-a9f8-40b9-811c-8c7808c1dab6 admin admin] PATH_INFO: `/v3/users/a013674911804dff8ed3c5b186d9abc7` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:04.340279 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-682e9a88-a9f8-40b9-811c-8c7808c1dab6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a013674911804dff8ed3c5b186d9abc7)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:04.340619 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb2e5cc-b308-4af3-8880-32920ec741f3 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:04.342281 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-682e9a88-a9f8-40b9-811c-8c7808c1dab6 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:04.380467 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:34:04.388540 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2cb2e5cc-b308-4af3-8880-32920ec741f3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '07896f1bd26c47d8bfed3b0ad7b7e90c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:04.391706 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-682e9a88-a9f8-40b9-811c-8c7808c1dab6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a013674911804dff8ed3c5b186d9abc7'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:04.396376 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2cb2e5cc-b308-4af3-8880-32920ec741f3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '07896f1bd26c47d8bfed3b0ad7b7e90c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:04.397113 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2cb2e5cc-b308-4af3-8880-32920ec741f3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '07896f1bd26c47d8bfed3b0ad7b7e90c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:04.398851 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-682e9a88-a9f8-40b9-811c-8c7808c1dab6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a013674911804dff8ed3c5b186d9abc7'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:04.405701 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:34:04.411529 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2cb2e5cc-b308-4af3-8880-32920ec741f3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '07896f1bd26c47d8bfed3b0ad7b7e90c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:04.413613 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-682e9a88-a9f8-40b9-811c-8c7808c1dab6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a013674911804dff8ed3c5b186d9abc7'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:04.414350 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-682e9a88-a9f8-40b9-811c-8c7808c1dab6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a013674911804dff8ed3c5b186d9abc7'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:04.414990 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-682e9a88-a9f8-40b9-811c-8c7808c1dab6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a013674911804dff8ed3c5b186d9abc7'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:04.421100 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2cb2e5cc-b308-4af3-8880-32920ec741f3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '07896f1bd26c47d8bfed3b0ad7b7e90c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:04.427914 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1136/2284] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:34:04 2022] DELETE /identity/v3/users/07896f1bd26c47d8bfed3b0ad7b7e90c => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:34:04.430990 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1149/2285] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:34:04 2022] DELETE /identity/v3/users/a013674911804dff8ed3c5b186d9abc7 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:34:04.432509 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1a4527-60a3-4cbf-90eb-f3d5af8306d0 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:04.432784 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1a4527-60a3-4cbf-90eb-f3d5af8306d0 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:04.432902 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59815712-9dd0-46cb-92c5-e38f9cda4b50 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:04.433048 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1a4527-60a3-4cbf-90eb-f3d5af8306d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:04.433178 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59815712-9dd0-46cb-92c5-e38f9cda4b50 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:04.433373 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59815712-9dd0-46cb-92c5-e38f9cda4b50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:04.452614 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-59815712-9dd0-46cb-92c5-e38f9cda4b50 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:34:04.453869 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-59815712-9dd0-46cb-92c5-e38f9cda4b50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:04.454071 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-7e1a4527-60a3-4cbf-90eb-f3d5af8306d0 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:34:04.455240 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-7e1a4527-60a3-4cbf-90eb-f3d5af8306d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:04.515501 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1137/2286] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:34:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:34:04.516824 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1150/2287] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:34:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:34:04.525688 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e1a4527-60a3-4cbf-90eb-f3d5af8306d0 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:04.526241 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59815712-9dd0-46cb-92c5-e38f9cda4b50 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:04.527114 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-7e1a4527-60a3-4cbf-90eb-f3d5af8306d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:04.527536 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-59815712-9dd0-46cb-92c5-e38f9cda4b50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:04.587310 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59815712-9dd0-46cb-92c5-e38f9cda4b50 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:04.587512 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e1a4527-60a3-4cbf-90eb-f3d5af8306d0 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:04.590882 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc46b0d-7403-4212-bd7b-c5c17a522395 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:04.591070 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a928856-7271-4b49-a3a1-3043cb09d0e9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:04.591654 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc46b0d-7403-4212-bd7b-c5c17a522395 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:04.591798 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a928856-7271-4b49-a3a1-3043cb09d0e9 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:04.591934 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc46b0d-7403-4212-bd7b-c5c17a522395 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:04.592095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a928856-7271-4b49-a3a1-3043cb09d0e9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:04.592208 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a928856-7271-4b49-a3a1-3043cb09d0e9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:04.592316 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc46b0d-7403-4212-bd7b-c5c17a522395 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:04.601898 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfc46b0d-7403-4212-bd7b-c5c17a522395 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:04.602044 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a928856-7271-4b49-a3a1-3043cb09d0e9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:04.603347 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfc46b0d-7403-4212-bd7b-c5c17a522395 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:04.603474 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a928856-7271-4b49-a3a1-3043cb09d0e9 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:04.612841 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1151/2288] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:34:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 27 01:34:04.613288 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1138/2289] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:34:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:34:04.891198 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc46b0d-7403-4212-bd7b-c5c17a522395 service neutron] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:04.892759 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-bfc46b0d-7403-4212-bd7b-c5c17a522395 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:04.908711 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc46b0d-7403-4212-bd7b-c5c17a522395 service neutron] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:04.914910 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880b3d1c-30b2-479c-adcc-944e5e28d965 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:04.915347 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880b3d1c-30b2-479c-adcc-944e5e28d965 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:04.915631 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880b3d1c-30b2-479c-adcc-944e5e28d965 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:04.915898 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880b3d1c-30b2-479c-adcc-944e5e28d965 admin admin] PATH_INFO: `/v3/projects/5c697d48969f44589359cbfb5952b34f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:04.917043 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-880b3d1c-30b2-479c-adcc-944e5e28d965 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5c697d48969f44589359cbfb5952b34f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:04.919028 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-880b3d1c-30b2-479c-adcc-944e5e28d965 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:04.922864 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a928856-7271-4b49-a3a1-3043cb09d0e9 service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:04.924252 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0a928856-7271-4b49-a3a1-3043cb09d0e9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:04.937629 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a928856-7271-4b49-a3a1-3043cb09d0e9 service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:04.944020 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7dcd0d-5cab-44e7-9711-b46635a88abc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:04.944639 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7dcd0d-5cab-44e7-9711-b46635a88abc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:04.944919 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7dcd0d-5cab-44e7-9711-b46635a88abc admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:04.945237 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7dcd0d-5cab-44e7-9711-b46635a88abc admin admin] PATH_INFO: `/v3/projects/0e3cfa878c494ae18fd619116fa25466` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:04.946377 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f7dcd0d-5cab-44e7-9711-b46635a88abc admin admin] RBAC: Authorizing `identity:delete_project(project_id=0e3cfa878c494ae18fd619116fa25466)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:04.948337 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f7dcd0d-5cab-44e7-9711-b46635a88abc admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:04.983921 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-880b3d1c-30b2-479c-adcc-944e5e28d965 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5c697d48969f44589359cbfb5952b34f', 'request_id': 'req-880b3d1c-30b2-479c-adcc-944e5e28d965'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:04.985657 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:34:04.994754 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-880b3d1c-30b2-479c-adcc-944e5e28d965 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5c697d48969f44589359cbfb5952b34f', 'request_id': 'req-880b3d1c-30b2-479c-adcc-944e5e28d965'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:05.001749 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-880b3d1c-30b2-479c-adcc-944e5e28d965 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5c697d48969f44589359cbfb5952b34f', 'request_id': 'req-880b3d1c-30b2-479c-adcc-944e5e28d965'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:05.001911 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9f7dcd0d-5cab-44e7-9711-b46635a88abc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0e3cfa878c494ae18fd619116fa25466', 'request_id': 'req-9f7dcd0d-5cab-44e7-9711-b46635a88abc'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:05.010846 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9f7dcd0d-5cab-44e7-9711-b46635a88abc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0e3cfa878c494ae18fd619116fa25466', 'request_id': 'req-9f7dcd0d-5cab-44e7-9711-b46635a88abc'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:05.013483 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-880b3d1c-30b2-479c-adcc-944e5e28d965 admin admin] The token cache is being invalidate because project 5c697d48969f44589359cbfb5952b34f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:34:05.013775 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-880b3d1c-30b2-479c-adcc-944e5e28d965 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:05.014410 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-880b3d1c-30b2-479c-adcc-944e5e28d965 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:05.016106 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1152/2290] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:34:04 2022] DELETE /identity/v3/projects/5c697d48969f44589359cbfb5952b34f => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:34:05.017284 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9f7dcd0d-5cab-44e7-9711-b46635a88abc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0e3cfa878c494ae18fd619116fa25466', 'request_id': 'req-9f7dcd0d-5cab-44e7-9711-b46635a88abc'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:05.021292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e8c511-d58b-460f-8d95-8982a89cff92 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:05.021604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e8c511-d58b-460f-8d95-8982a89cff92 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:05.021881 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e8c511-d58b-460f-8d95-8982a89cff92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:05.029864 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9f7dcd0d-5cab-44e7-9711-b46635a88abc admin admin] The token cache is being invalidate because project 0e3cfa878c494ae18fd619116fa25466 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:34:05.030162 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9f7dcd0d-5cab-44e7-9711-b46635a88abc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:05.030938 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9f7dcd0d-5cab-44e7-9711-b46635a88abc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:05.032568 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1139/2291] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:34:04 2022] DELETE /identity/v3/projects/0e3cfa878c494ae18fd619116fa25466 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:34:05.037332 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7dcd0d-5cab-44e7-9711-b46635a88abc admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:05.038659 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-9f7dcd0d-5cab-44e7-9711-b46635a88abc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:05.042561 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-61e8c511-d58b-460f-8d95-8982a89cff92 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:34:05.043730 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-61e8c511-d58b-460f-8d95-8982a89cff92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:05.053600 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:34:05.107751 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1153/2292] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:34:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:34:05.111977 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7dcd0d-5cab-44e7-9711-b46635a88abc admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:05.115857 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1071575-a50f-40a1-aef2-f90d4b97017f None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:05.116179 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1071575-a50f-40a1-aef2-f90d4b97017f None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:05.116422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1071575-a50f-40a1-aef2-f90d4b97017f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:05.117809 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d37471-f964-4097-9092-5ff4d4621ed7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:05.118440 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d37471-f964-4097-9092-5ff4d4621ed7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:05.118717 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d37471-f964-4097-9092-5ff4d4621ed7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:05.118990 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d37471-f964-4097-9092-5ff4d4621ed7 admin admin] PATH_INFO: `/v3/users/fd38cadd45944b428959df77c29148ea` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:05.120163 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d37471-f964-4097-9092-5ff4d4621ed7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fd38cadd45944b428959df77c29148ea)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:05.122146 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d37471-f964-4097-9092-5ff4d4621ed7 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:05.136876 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-c1071575-a50f-40a1-aef2-f90d4b97017f None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:34:05.138149 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-c1071575-a50f-40a1-aef2-f90d4b97017f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:05.146190 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1154/2293] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:34:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:34:05.150881 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1071575-a50f-40a1-aef2-f90d4b97017f None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:05.160438 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1071575-a50f-40a1-aef2-f90d4b97017f None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:05.171078 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa48bc3-a8d2-4195-be82-fced436fb3f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:05.171696 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa48bc3-a8d2-4195-be82-fced436fb3f0 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:05.171970 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa48bc3-a8d2-4195-be82-fced436fb3f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:05.172234 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa48bc3-a8d2-4195-be82-fced436fb3f0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:05.173190 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caa48bc3-a8d2-4195-be82-fced436fb3f0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81588) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:34:05.173492 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caa48bc3-a8d2-4195-be82-fced436fb3f0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:05.174001 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-a1d37471-f964-4097-9092-5ff4d4621ed7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fd38cadd45944b428959df77c29148ea'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:05.175086 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caa48bc3-a8d2-4195-be82-fced436fb3f0 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:05.183787 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-a1d37471-f964-4097-9092-5ff4d4621ed7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fd38cadd45944b428959df77c29148ea'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:05.184470 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-a1d37471-f964-4097-9092-5ff4d4621ed7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fd38cadd45944b428959df77c29148ea'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:05.189337 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1155/2294] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:34:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:34:05.194451 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa48bc3-a8d2-4195-be82-fced436fb3f0 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:05.196143 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-caa48bc3-a8d2-4195-be82-fced436fb3f0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:05.197119 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-a1d37471-f964-4097-9092-5ff4d4621ed7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fd38cadd45944b428959df77c29148ea'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:05.203891 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-a1d37471-f964-4097-9092-5ff4d4621ed7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fd38cadd45944b428959df77c29148ea'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:05.209292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa48bc3-a8d2-4195-be82-fced436fb3f0 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:05.211086 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1140/2295] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:34:05 2022] DELETE /identity/v3/users/fd38cadd45944b428959df77c29148ea => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:34:05.269859 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d838586-21f7-4044-9c1a-15286e9c1680 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:05.272027 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d838586-21f7-4044-9c1a-15286e9c1680 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:05.272311 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d838586-21f7-4044-9c1a-15286e9c1680 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:05.272579 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d838586-21f7-4044-9c1a-15286e9c1680 admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:05.273246 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d838586-21f7-4044-9c1a-15286e9c1680 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:05.275189 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d838586-21f7-4044-9c1a-15286e9c1680 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:05.306694 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1156/2296] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:34:05 2022] POST /identity/v3/projects => generated 383 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:34:05.310832 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d37471-f964-4097-9092-5ff4d4621ed7 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:05.321702 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d37471-f964-4097-9092-5ff4d4621ed7 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:05.385451 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-780be4f3-e551-4848-b94d-0a58c42ff592 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:05.386152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780be4f3-e551-4848-b94d-0a58c42ff592 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:05.386496 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780be4f3-e551-4848-b94d-0a58c42ff592 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:05.386820 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780be4f3-e551-4848-b94d-0a58c42ff592 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:05.387540 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-780be4f3-e551-4848-b94d-0a58c42ff592 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:05.389460 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-780be4f3-e551-4848-b94d-0a58c42ff592 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:05.412946 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1141/2297] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:34:05 2022] POST /identity/v3/users => generated 292 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:34:05.417004 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d838586-21f7-4044-9c1a-15286e9c1680 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:05.427487 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d838586-21f7-4044-9c1a-15286e9c1680 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:05.431919 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661220ac-1c3b-44f2-98dd-29f3a7093a95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:05.432512 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661220ac-1c3b-44f2-98dd-29f3a7093a95 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:05.432792 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661220ac-1c3b-44f2-98dd-29f3a7093a95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:05.433073 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661220ac-1c3b-44f2-98dd-29f3a7093a95 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:05.433799 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-661220ac-1c3b-44f2-98dd-29f3a7093a95 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:05.435519 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-661220ac-1c3b-44f2-98dd-29f3a7093a95 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:05.449169 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1157/2298] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:34:05 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:34:05.450308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-780be4f3-e551-4848-b94d-0a58c42ff592 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:05.460750 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-780be4f3-e551-4848-b94d-0a58c42ff592 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:05.465448 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b956ff-2b46-463d-8855-8c85d5b37934 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:05.466049 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b956ff-2b46-463d-8855-8c85d5b37934 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:05.466324 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b956ff-2b46-463d-8855-8c85d5b37934 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:05.466588 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b956ff-2b46-463d-8855-8c85d5b37934 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:05.467227 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b956ff-2b46-463d-8855-8c85d5b37934 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:05.468820 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b956ff-2b46-463d-8855-8c85d5b37934 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:05.479539 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1142/2299] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:34:05 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:34:05.484004 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661220ac-1c3b-44f2-98dd-29f3a7093a95 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:05.497848 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661220ac-1c3b-44f2-98dd-29f3a7093a95 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:05.499053 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b956ff-2b46-463d-8855-8c85d5b37934 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:05.501703 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-b5b956ff-2b46-463d-8855-8c85d5b37934 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:05.503640 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe54545-7abe-475e-a70f-d03f780daab3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:05.504683 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe54545-7abe-475e-a70f-d03f780daab3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:05.505276 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe54545-7abe-475e-a70f-d03f780daab3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:05.505557 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe54545-7abe-475e-a70f-d03f780daab3 admin admin] PATH_INFO: `/v3/projects/176e5a3c112e4967a83375c948008ab5/users/2321134e938247b09711047a12b86382/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:05.516249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b956ff-2b46-463d-8855-8c85d5b37934 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:05.522557 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe54545-7abe-475e-a70f-d03f780daab3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=176e5a3c112e4967a83375c948008ab5, user_id=2321134e938247b09711047a12b86382, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:05.523933 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334f1cb7-d1cf-46bc-a5be-244e8ec4af27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:05.524587 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334f1cb7-d1cf-46bc-a5be-244e8ec4af27 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:05.524862 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334f1cb7-d1cf-46bc-a5be-244e8ec4af27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:05.525163 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334f1cb7-d1cf-46bc-a5be-244e8ec4af27 admin admin] PATH_INFO: `/v3/projects/919490995bfa4c2b8084cfbcdb2dcb01` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:05.525292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe54545-7abe-475e-a70f-d03f780daab3 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:05.526469 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-334f1cb7-d1cf-46bc-a5be-244e8ec4af27 admin admin] RBAC: Authorizing `identity:delete_project(project_id=919490995bfa4c2b8084cfbcdb2dcb01)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:05.529002 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-334f1cb7-d1cf-46bc-a5be-244e8ec4af27 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:05.531870 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:34:05.542843 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1158/2300] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:34:05 2022] PUT /identity/v3/projects/176e5a3c112e4967a83375c948008ab5/users/2321134e938247b09711047a12b86382/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:34:05.547943 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f14b01f-f357-4bba-9b05-5885b093a629 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:05.548222 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f14b01f-f357-4bba-9b05-5885b093a629 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:05.548516 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f14b01f-f357-4bba-9b05-5885b093a629 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:05.569464 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-0f14b01f-f357-4bba-9b05-5885b093a629 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:34:05.570670 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-0f14b01f-f357-4bba-9b05-5885b093a629 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:05.574833 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:INSERT=3,keystone:DELETE=3,keystone:SELECT=94 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:34:05.582535 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-334f1cb7-d1cf-46bc-a5be-244e8ec4af27 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '919490995bfa4c2b8084cfbcdb2dcb01', 'request_id': 'req-334f1cb7-d1cf-46bc-a5be-244e8ec4af27'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:05.592120 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-334f1cb7-d1cf-46bc-a5be-244e8ec4af27 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '919490995bfa4c2b8084cfbcdb2dcb01', 'request_id': 'req-334f1cb7-d1cf-46bc-a5be-244e8ec4af27'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:05.599439 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-334f1cb7-d1cf-46bc-a5be-244e8ec4af27 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '919490995bfa4c2b8084cfbcdb2dcb01', 'request_id': 'req-334f1cb7-d1cf-46bc-a5be-244e8ec4af27'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:05.611664 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-334f1cb7-d1cf-46bc-a5be-244e8ec4af27 admin admin] The token cache is being invalidate because project 919490995bfa4c2b8084cfbcdb2dcb01 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:34:05.611664 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-334f1cb7-d1cf-46bc-a5be-244e8ec4af27 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:05.612257 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-334f1cb7-d1cf-46bc-a5be-244e8ec4af27 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:05.614675 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1143/2301] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:34:05 2022] DELETE /identity/v3/projects/919490995bfa4c2b8084cfbcdb2dcb01 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:34:05.618331 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334f1cb7-d1cf-46bc-a5be-244e8ec4af27 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:05.619761 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-334f1cb7-d1cf-46bc-a5be-244e8ec4af27 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:05.638241 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1159/2302] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:34:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:34:05.649954 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f14b01f-f357-4bba-9b05-5885b093a629 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:05.651437 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-0f14b01f-f357-4bba-9b05-5885b093a629 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:05.693621 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334f1cb7-d1cf-46bc-a5be-244e8ec4af27 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:05.699034 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a2f70a-d918-4e1f-a044-712ce4c141e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:05.699655 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a2f70a-d918-4e1f-a044-712ce4c141e2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:05.699931 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a2f70a-d918-4e1f-a044-712ce4c141e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:05.700197 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a2f70a-d918-4e1f-a044-712ce4c141e2 admin admin] PATH_INFO: `/v3/users/c031fd72c2104217bf4b9a7587ad438b` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:05.701352 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a2f70a-d918-4e1f-a044-712ce4c141e2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c031fd72c2104217bf4b9a7587ad438b)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:05.703272 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a2f70a-d918-4e1f-a044-712ce4c141e2 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:05.710190 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f14b01f-f357-4bba-9b05-5885b093a629 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:05.714563 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-476e63f3-f83c-4c45-a2a6-9d1dc55e8d44 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:05.715796 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476e63f3-f83c-4c45-a2a6-9d1dc55e8d44 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:05.716254 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476e63f3-f83c-4c45-a2a6-9d1dc55e8d44 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:05.716697 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476e63f3-f83c-4c45-a2a6-9d1dc55e8d44 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:05.728531 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-476e63f3-f83c-4c45-a2a6-9d1dc55e8d44 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:05.730268 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-476e63f3-f83c-4c45-a2a6-9d1dc55e8d44 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:05.742676 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1160/2303] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:34:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:34:05.755749 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-82a2f70a-d918-4e1f-a044-712ce4c141e2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c031fd72c2104217bf4b9a7587ad438b'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:05.765753 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-82a2f70a-d918-4e1f-a044-712ce4c141e2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c031fd72c2104217bf4b9a7587ad438b'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:05.766642 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-82a2f70a-d918-4e1f-a044-712ce4c141e2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c031fd72c2104217bf4b9a7587ad438b'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:05.781336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-82a2f70a-d918-4e1f-a044-712ce4c141e2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c031fd72c2104217bf4b9a7587ad438b'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:05.789512 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-82a2f70a-d918-4e1f-a044-712ce4c141e2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c031fd72c2104217bf4b9a7587ad438b'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:05.795420 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1144/2304] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:34:05 2022] DELETE /identity/v3/users/c031fd72c2104217bf4b9a7587ad438b => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:34:06.178180 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-476e63f3-f83c-4c45-a2a6-9d1dc55e8d44 service neutron] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:06.192123 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-476e63f3-f83c-4c45-a2a6-9d1dc55e8d44 service neutron] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:06.260189 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8710a6-d818-4aeb-a4df-3c36e6d479a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:06.260857 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8710a6-d818-4aeb-a4df-3c36e6d479a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:06.261166 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8710a6-d818-4aeb-a4df-3c36e6d479a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:06.261441 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8710a6-d818-4aeb-a4df-3c36e6d479a0 admin admin] PATH_INFO: `/v3/projects/1767f1f9014b4a6e99e343757adf03cf` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:06.262563 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8710a6-d818-4aeb-a4df-3c36e6d479a0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1767f1f9014b4a6e99e343757adf03cf)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:06.264454 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8710a6-d818-4aeb-a4df-3c36e6d479a0 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:06.312989 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-fa8710a6-d818-4aeb-a4df-3c36e6d479a0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1767f1f9014b4a6e99e343757adf03cf', 'request_id': 'req-fa8710a6-d818-4aeb-a4df-3c36e6d479a0'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:06.316511 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:34:06.322758 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-fa8710a6-d818-4aeb-a4df-3c36e6d479a0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1767f1f9014b4a6e99e343757adf03cf', 'request_id': 'req-fa8710a6-d818-4aeb-a4df-3c36e6d479a0'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:06.330273 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-fa8710a6-d818-4aeb-a4df-3c36e6d479a0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1767f1f9014b4a6e99e343757adf03cf', 'request_id': 'req-fa8710a6-d818-4aeb-a4df-3c36e6d479a0'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:06.341900 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-fa8710a6-d818-4aeb-a4df-3c36e6d479a0 admin admin] The token cache is being invalidate because project 1767f1f9014b4a6e99e343757adf03cf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:34:06.342244 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-fa8710a6-d818-4aeb-a4df-3c36e6d479a0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:06.342890 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-fa8710a6-d818-4aeb-a4df-3c36e6d479a0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:06.344408 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1161/2305] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:34:06 2022] DELETE /identity/v3/projects/1767f1f9014b4a6e99e343757adf03cf => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:34:06.350837 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4415ccb4-95b1-47f1-87e0-0fc984861114 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:06.351338 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4415ccb4-95b1-47f1-87e0-0fc984861114 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:06.351453 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4415ccb4-95b1-47f1-87e0-0fc984861114 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:06.372167 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-4415ccb4-95b1-47f1-87e0-0fc984861114 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:34:06.373585 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-4415ccb4-95b1-47f1-87e0-0fc984861114 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:06.379311 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:34:06.455002 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1145/2306] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:34:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:34:06.469542 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd51314-9dc8-4f8d-b9e1-2bf10bfa8931 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:06.469542 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd51314-9dc8-4f8d-b9e1-2bf10bfa8931 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:06.469542 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd51314-9dc8-4f8d-b9e1-2bf10bfa8931 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:06.474308 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b193ec-d3f3-46e1-9c84-c1da4d66fc6e None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:06.474592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b193ec-d3f3-46e1-9c84-c1da4d66fc6e None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:06.474865 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b193ec-d3f3-46e1-9c84-c1da4d66fc6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:06.498982 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-d3b193ec-d3f3-46e1-9c84-c1da4d66fc6e None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:34:06.500785 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d3b193ec-d3f3-46e1-9c84-c1da4d66fc6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:06.503075 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-4cd51314-9dc8-4f8d-b9e1-2bf10bfa8931 None None] MFA Rules not processed for user `2321134e938247b09711047a12b86382`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:34:06.504302 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-4cd51314-9dc8-4f8d-b9e1-2bf10bfa8931 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:06.511583 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1146/2307] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:34:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:34:06.516166 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b193ec-d3f3-46e1-9c84-c1da4d66fc6e None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:06.528566 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b193ec-d3f3-46e1-9c84-c1da4d66fc6e None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:06.533759 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c37f9e-014d-4d27-91a6-ec9beb78170e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:06.535014 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c37f9e-014d-4d27-91a6-ec9beb78170e admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:06.535289 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c37f9e-014d-4d27-91a6-ec9beb78170e admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:06.535557 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c37f9e-014d-4d27-91a6-ec9beb78170e admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:06.536813 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c37f9e-014d-4d27-91a6-ec9beb78170e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:34:06.537312 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c37f9e-014d-4d27-91a6-ec9beb78170e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:06.539967 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c37f9e-014d-4d27-91a6-ec9beb78170e admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:06.553106 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1147/2308] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:34:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:34:06.557141 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c37f9e-014d-4d27-91a6-ec9beb78170e admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:06.569092 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c37f9e-014d-4d27-91a6-ec9beb78170e admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:06.573983 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc967ba1-d4ae-4fb4-b73d-c32362c45c00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:06.574677 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc967ba1-d4ae-4fb4-b73d-c32362c45c00 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:06.575063 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc967ba1-d4ae-4fb4-b73d-c32362c45c00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:06.575207 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc967ba1-d4ae-4fb4-b73d-c32362c45c00 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:06.576016 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc967ba1-d4ae-4fb4-b73d-c32362c45c00 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:06.577999 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc967ba1-d4ae-4fb4-b73d-c32362c45c00 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:06.587113 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1162/2309] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:34:06 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:34:06.597973 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd51314-9dc8-4f8d-b9e1-2bf10bfa8931 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:06.599575 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-4cd51314-9dc8-4f8d-b9e1-2bf10bfa8931 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:06.611978 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1148/2310] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:34:06 2022] POST /identity/v3/projects => generated 393 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:34:06.616197 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc967ba1-d4ae-4fb4-b73d-c32362c45c00 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:06.668822 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd51314-9dc8-4f8d-b9e1-2bf10bfa8931 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:06.682241 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc967ba1-d4ae-4fb4-b73d-c32362c45c00 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:06.687146 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c65daf-9b0d-4277-af00-6d5cfd533329 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:06.687753 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c65daf-9b0d-4277-af00-6d5cfd533329 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:06.688027 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c65daf-9b0d-4277-af00-6d5cfd533329 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:06.688533 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c65daf-9b0d-4277-af00-6d5cfd533329 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:06.689211 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c65daf-9b0d-4277-af00-6d5cfd533329 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:06.691230 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c65daf-9b0d-4277-af00-6d5cfd533329 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:06.719782 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1149/2311] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:34:06 2022] POST /identity/v3/users => generated 297 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:34:06.724119 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c65daf-9b0d-4277-af00-6d5cfd533329 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:06.726166 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39139ad9-e9d6-4f94-8d5e-82274068afb3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:06.726778 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39139ad9-e9d6-4f94-8d5e-82274068afb3 service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:06.727056 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39139ad9-e9d6-4f94-8d5e-82274068afb3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:06.727326 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39139ad9-e9d6-4f94-8d5e-82274068afb3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:06.735931 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c65daf-9b0d-4277-af00-6d5cfd533329 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:06.742660 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535935e1-92f5-4718-afb0-96f4ee55027a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:06.743258 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535935e1-92f5-4718-afb0-96f4ee55027a admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:06.743534 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535935e1-92f5-4718-afb0-96f4ee55027a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:06.743799 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535935e1-92f5-4718-afb0-96f4ee55027a admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:06.744447 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-535935e1-92f5-4718-afb0-96f4ee55027a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:06.746081 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-535935e1-92f5-4718-afb0-96f4ee55027a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:06.760112 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1150/2312] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:34:06 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 27 01:34:06.761202 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535935e1-92f5-4718-afb0-96f4ee55027a admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:06.771836 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535935e1-92f5-4718-afb0-96f4ee55027a admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:06.776696 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87230697-5426-4111-8cec-0ae2b2a960aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:06.777323 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87230697-5426-4111-8cec-0ae2b2a960aa admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:06.777602 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87230697-5426-4111-8cec-0ae2b2a960aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:06.777874 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87230697-5426-4111-8cec-0ae2b2a960aa admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:06.778510 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87230697-5426-4111-8cec-0ae2b2a960aa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:06.780151 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87230697-5426-4111-8cec-0ae2b2a960aa admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:06.787293 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:34:06.793547 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1151/2313] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:34:06 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:34:06.798226 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87230697-5426-4111-8cec-0ae2b2a960aa admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:06.810788 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87230697-5426-4111-8cec-0ae2b2a960aa admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:06.817379 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58db4202-9d64-485d-afbc-e710df2a4557 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:06.818446 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58db4202-9d64-485d-afbc-e710df2a4557 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:06.819249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58db4202-9d64-485d-afbc-e710df2a4557 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:06.819249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58db4202-9d64-485d-afbc-e710df2a4557 admin admin] PATH_INFO: `/v3/projects/105497a27f944bfdb9e844ffb00ea024/users/34c5466f204c4e64ace74e7ce326f73e/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:06.821271 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39139ad9-e9d6-4f94-8d5e-82274068afb3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:06.822866 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39139ad9-e9d6-4f94-8d5e-82274068afb3 service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:06.832419 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1163/2314] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:34:06 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 235 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 27 01:34:06.836267 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58db4202-9d64-485d-afbc-e710df2a4557 admin admin] RBAC: Authorizing `identity:create_grant(project_id=105497a27f944bfdb9e844ffb00ea024, user_id=34c5466f204c4e64ace74e7ce326f73e, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:06.838948 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58db4202-9d64-485d-afbc-e710df2a4557 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:06.853093 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1152/2315] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:34:06 2022] PUT /identity/v3/projects/105497a27f944bfdb9e844ffb00ea024/users/34c5466f204c4e64ace74e7ce326f73e/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:34:06.865241 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624c9d6f-335f-423f-8196-4d55e7ce7d2e None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:06.865525 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624c9d6f-335f-423f-8196-4d55e7ce7d2e None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:06.865824 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624c9d6f-335f-423f-8196-4d55e7ce7d2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:06.895078 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-624c9d6f-335f-423f-8196-4d55e7ce7d2e None None] MFA Rules not processed for user `34c5466f204c4e64ace74e7ce326f73e`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:34:06.896374 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-624c9d6f-335f-423f-8196-4d55e7ce7d2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:06.964822 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1164/2316] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:34:06 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:34:06.969503 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58db4202-9d64-485d-afbc-e710df2a4557 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:06.982695 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:34:07.035789 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58db4202-9d64-485d-afbc-e710df2a4557 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:07.041821 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed02c3e5-e95c-4c5e-a9e3-b8ee4d93ca9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:07.042683 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed02c3e5-e95c-4c5e-a9e3-b8ee4d93ca9e admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:07.043017 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed02c3e5-e95c-4c5e-a9e3-b8ee4d93ca9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:07.043356 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed02c3e5-e95c-4c5e-a9e3-b8ee4d93ca9e admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:07.044160 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed02c3e5-e95c-4c5e-a9e3-b8ee4d93ca9e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:07.046274 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed02c3e5-e95c-4c5e-a9e3-b8ee4d93ca9e admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:07.077806 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1153/2317] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:34:06 2022] POST /identity/v3/projects => generated 391 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:34:07.091786 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-624c9d6f-335f-423f-8196-4d55e7ce7d2e None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:07.093795 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-624c9d6f-335f-423f-8196-4d55e7ce7d2e None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:07.178824 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18116017-c34b-4fed-8459-9dd3f11add95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:07.179440 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18116017-c34b-4fed-8459-9dd3f11add95 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:07.179717 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18116017-c34b-4fed-8459-9dd3f11add95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:07.180109 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18116017-c34b-4fed-8459-9dd3f11add95 admin admin] PATH_INFO: `/v3/users` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:07.180640 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18116017-c34b-4fed-8459-9dd3f11add95 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:07.182575 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18116017-c34b-4fed-8459-9dd3f11add95 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:07.204428 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1165/2318] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:34:07 2022] POST /identity/v3/users => generated 302 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:34:07.208492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed02c3e5-e95c-4c5e-a9e3-b8ee4d93ca9e admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:07.218528 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed02c3e5-e95c-4c5e-a9e3-b8ee4d93ca9e admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:07.223250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede4e0cf-6f99-42ce-b8fc-3431aaeff750 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:07.224416 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede4e0cf-6f99-42ce-b8fc-3431aaeff750 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:07.224693 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede4e0cf-6f99-42ce-b8fc-3431aaeff750 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:07.224964 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede4e0cf-6f99-42ce-b8fc-3431aaeff750 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:07.225668 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ede4e0cf-6f99-42ce-b8fc-3431aaeff750 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:07.227293 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ede4e0cf-6f99-42ce-b8fc-3431aaeff750 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:07.238139 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1154/2319] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:34:07 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:34:07.242848 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18116017-c34b-4fed-8459-9dd3f11add95 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:07.252905 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18116017-c34b-4fed-8459-9dd3f11add95 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:07.257690 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5958fe67-22d6-437e-ba21-40125d342069 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:07.258378 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5958fe67-22d6-437e-ba21-40125d342069 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:07.258633 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5958fe67-22d6-437e-ba21-40125d342069 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:07.258902 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5958fe67-22d6-437e-ba21-40125d342069 admin admin] PATH_INFO: `/v3/projects/81f90a474e434ed6ab3ed1b03556415e/users/18b360d4555d4f3c9fd64e35ee90f9c5/roles/414286927fdd44688107ff5e10235acf` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:07.273475 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5958fe67-22d6-437e-ba21-40125d342069 admin admin] RBAC: Authorizing `identity:create_grant(project_id=81f90a474e434ed6ab3ed1b03556415e, user_id=18b360d4555d4f3c9fd64e35ee90f9c5, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:07.276126 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5958fe67-22d6-437e-ba21-40125d342069 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:07.289663 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1166/2320] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:34:07 2022] PUT /identity/v3/projects/81f90a474e434ed6ab3ed1b03556415e/users/18b360d4555d4f3c9fd64e35ee90f9c5/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:34:07.293559 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede4e0cf-6f99-42ce-b8fc-3431aaeff750 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:07.306243 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede4e0cf-6f99-42ce-b8fc-3431aaeff750 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:07.364891 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6ff24a-9484-4293-9e63-2dfe1dfb3bb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:07.365608 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6ff24a-9484-4293-9e63-2dfe1dfb3bb0 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:07.365790 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6ff24a-9484-4293-9e63-2dfe1dfb3bb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:07.366063 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6ff24a-9484-4293-9e63-2dfe1dfb3bb0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:07.366700 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6ff24a-9484-4293-9e63-2dfe1dfb3bb0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:07.368305 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6ff24a-9484-4293-9e63-2dfe1dfb3bb0 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:07.377979 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1155/2321] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:34:07 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:34:07.382223 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5958fe67-22d6-437e-ba21-40125d342069 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:07.392264 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5958fe67-22d6-437e-ba21-40125d342069 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:07.396935 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994558f2-aed8-4f96-8758-ecdc814244ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:07.397605 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994558f2-aed8-4f96-8758-ecdc814244ff admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:07.397882 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994558f2-aed8-4f96-8758-ecdc814244ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:07.398152 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994558f2-aed8-4f96-8758-ecdc814244ff admin admin] PATH_INFO: `/v3/projects/81f90a474e434ed6ab3ed1b03556415e/users/18b360d4555d4f3c9fd64e35ee90f9c5/roles/414286927fdd44688107ff5e10235acf` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:07.400022 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994558f2-aed8-4f96-8758-ecdc814244ff admin admin] RBAC: Authorizing `identity:create_grant(project_id=81f90a474e434ed6ab3ed1b03556415e, user_id=18b360d4555d4f3c9fd64e35ee90f9c5, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:07.402680 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994558f2-aed8-4f96-8758-ecdc814244ff admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:07.413411 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1167/2322] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:34:07 2022] PUT /identity/v3/projects/81f90a474e434ed6ab3ed1b03556415e/users/18b360d4555d4f3c9fd64e35ee90f9c5/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:34:07.424758 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53123296-95df-47b1-a3cc-3a32f212b37d None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:07.425039 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53123296-95df-47b1-a3cc-3a32f212b37d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:07.425345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53123296-95df-47b1-a3cc-3a32f212b37d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:07.453288 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-53123296-95df-47b1-a3cc-3a32f212b37d None None] MFA Rules not processed for user `18b360d4555d4f3c9fd64e35ee90f9c5`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:34:07.454422 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-53123296-95df-47b1-a3cc-3a32f212b37d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:07.521660 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1156/2323] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:34:07 2022] POST /identity/v3/auth/tokens => generated 3335 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:34:07.532820 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994558f2-aed8-4f96-8758-ecdc814244ff admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:07.560740 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:34:07.590306 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994558f2-aed8-4f96-8758-ecdc814244ff admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:07.593922 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b397466a-365c-4e0b-a728-4cf8b322697a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:07.594512 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b397466a-365c-4e0b-a728-4cf8b322697a service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:07.594783 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b397466a-365c-4e0b-a728-4cf8b322697a service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:07.595052 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b397466a-365c-4e0b-a728-4cf8b322697a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:07.607876 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b397466a-365c-4e0b-a728-4cf8b322697a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:07.609307 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b397466a-365c-4e0b-a728-4cf8b322697a service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:07.619419 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1168/2324] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:34:07 2022] GET /identity/v3/auth/tokens => generated 3335 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:34:07.666741 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53123296-95df-47b1-a3cc-3a32f212b37d None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:07.668112 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-53123296-95df-47b1-a3cc-3a32f212b37d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:07.680771 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53123296-95df-47b1-a3cc-3a32f212b37d None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:07.684333 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510c4688-9ef7-4574-8688-59a74c733f59 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:07.684915 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510c4688-9ef7-4574-8688-59a74c733f59 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:07.685224 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510c4688-9ef7-4574-8688-59a74c733f59 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:07.685494 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510c4688-9ef7-4574-8688-59a74c733f59 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:07.689476 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:34:07.743283 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-510c4688-9ef7-4574-8688-59a74c733f59 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:07.744629 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-510c4688-9ef7-4574-8688-59a74c733f59 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:07.753233 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1157/2325] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:34:07 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:34:08.214647 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3475ad-e1bf-421b-9d5c-cd56c8bd501d None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:08.214914 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3475ad-e1bf-421b-9d5c-cd56c8bd501d None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:08.215185 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3475ad-e1bf-421b-9d5c-cd56c8bd501d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:08.234353 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-2e3475ad-e1bf-421b-9d5c-cd56c8bd501d None None] MFA Rules not processed for user `2321134e938247b09711047a12b86382`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:34:08.235553 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-2e3475ad-e1bf-421b-9d5c-cd56c8bd501d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:08.289796 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1169/2326] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:34:08 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:34:08.328973 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510c4688-9ef7-4574-8688-59a74c733f59 service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:08.330455 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-510c4688-9ef7-4574-8688-59a74c733f59 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:08.387842 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510c4688-9ef7-4574-8688-59a74c733f59 service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:08.391433 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfaff480-6c00-437a-a25d-227f11feb831 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2880dbb00eec471285d242e8183dd8d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:08.392027 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaff480-6c00-437a-a25d-227f11feb831 service glance] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:08.392303 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaff480-6c00-437a-a25d-227f11feb831 service glance] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:08.392571 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaff480-6c00-437a-a25d-227f11feb831 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:08.394271 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-bfaff480-6c00-437a-a25d-227f11feb831 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:08.463306 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfaff480-6c00-437a-a25d-227f11feb831 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:08.464725 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfaff480-6c00-437a-a25d-227f11feb831 service glance] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:08.476061 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1158/2327] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:34:08 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:34:09.005685 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3475ad-e1bf-421b-9d5c-cd56c8bd501d None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:09.007229 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-2e3475ad-e1bf-421b-9d5c-cd56c8bd501d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:09.068649 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3475ad-e1bf-421b-9d5c-cd56c8bd501d None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:09.072007 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c4a594f-b8bb-47ca-93c0-211a2f0291cd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:09.072687 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4a594f-b8bb-47ca-93c0-211a2f0291cd service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:09.073085 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4a594f-b8bb-47ca-93c0-211a2f0291cd service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:09.073544 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4a594f-b8bb-47ca-93c0-211a2f0291cd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:09.075583 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-3c4a594f-b8bb-47ca-93c0-211a2f0291cd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:09.090519 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c4a594f-b8bb-47ca-93c0-211a2f0291cd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:09.091982 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c4a594f-b8bb-47ca-93c0-211a2f0291cd service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:09.112347 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1170/2328] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:34:09 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:34:11.091353 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfaff480-6c00-437a-a25d-227f11feb831 service glance] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:11.141109 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:34:11.157023 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfaff480-6c00-437a-a25d-227f11feb831 service glance] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:11.161906 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b016f86-3f27-40c7-8321-a06c15eaa078 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:11.162544 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b016f86-3f27-40c7-8321-a06c15eaa078 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:11.162829 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b016f86-3f27-40c7-8321-a06c15eaa078 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:11.163104 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b016f86-3f27-40c7-8321-a06c15eaa078 admin admin] PATH_INFO: `/v3/users/34c5466f204c4e64ace74e7ce326f73e` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:11.164241 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b016f86-3f27-40c7-8321-a06c15eaa078 admin admin] RBAC: Authorizing `identity:delete_user(user_id=34c5466f204c4e64ace74e7ce326f73e)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:11.166194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b016f86-3f27-40c7-8321-a06c15eaa078 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:11.215129 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6b016f86-3f27-40c7-8321-a06c15eaa078 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '34c5466f204c4e64ace74e7ce326f73e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:11.224622 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6b016f86-3f27-40c7-8321-a06c15eaa078 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '34c5466f204c4e64ace74e7ce326f73e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:11.225344 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6b016f86-3f27-40c7-8321-a06c15eaa078 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '34c5466f204c4e64ace74e7ce326f73e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:11.237793 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6b016f86-3f27-40c7-8321-a06c15eaa078 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '34c5466f204c4e64ace74e7ce326f73e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:11.246061 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6b016f86-3f27-40c7-8321-a06c15eaa078 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '34c5466f204c4e64ace74e7ce326f73e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:11.253475 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1159/2329] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:34:11 2022] DELETE /identity/v3/users/34c5466f204c4e64ace74e7ce326f73e => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:34:11.258461 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08301f1-6172-425f-9475-39e5889effea None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:11.259003 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08301f1-6172-425f-9475-39e5889effea None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:11.259239 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08301f1-6172-425f-9475-39e5889effea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:11.278852 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-d08301f1-6172-425f-9475-39e5889effea None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:34:11.280149 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-d08301f1-6172-425f-9475-39e5889effea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:11.355009 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1171/2330] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:34:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:34:11.366284 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b016f86-3f27-40c7-8321-a06c15eaa078 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:11.367775 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-6b016f86-3f27-40c7-8321-a06c15eaa078 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:11.431607 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b016f86-3f27-40c7-8321-a06c15eaa078 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:11.435461 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4f17ceb-2322-4c39-8598-e31bed555af6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:11.436070 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f17ceb-2322-4c39-8598-e31bed555af6 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:11.436351 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f17ceb-2322-4c39-8598-e31bed555af6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:11.436622 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f17ceb-2322-4c39-8598-e31bed555af6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:11.446901 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4f17ceb-2322-4c39-8598-e31bed555af6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:11.449358 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4f17ceb-2322-4c39-8598-e31bed555af6 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:11.460388 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1160/2331] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:34:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:34:11.853821 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08301f1-6172-425f-9475-39e5889effea None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:11.855308 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-d08301f1-6172-425f-9475-39e5889effea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:11.875601 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08301f1-6172-425f-9475-39e5889effea None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:11.880352 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e97693-1ee1-43ba-83ff-bd01088350f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:11.880991 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e97693-1ee1-43ba-83ff-bd01088350f7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:11.881315 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e97693-1ee1-43ba-83ff-bd01088350f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:11.881589 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e97693-1ee1-43ba-83ff-bd01088350f7 admin admin] PATH_INFO: `/v3/projects/105497a27f944bfdb9e844ffb00ea024` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:11.882746 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e97693-1ee1-43ba-83ff-bd01088350f7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=105497a27f944bfdb9e844ffb00ea024)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:11.884664 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e97693-1ee1-43ba-83ff-bd01088350f7 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:11.895818 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:34:11.934746 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-e4e97693-1ee1-43ba-83ff-bd01088350f7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '105497a27f944bfdb9e844ffb00ea024', 'request_id': 'req-e4e97693-1ee1-43ba-83ff-bd01088350f7'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:11.943681 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-e4e97693-1ee1-43ba-83ff-bd01088350f7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '105497a27f944bfdb9e844ffb00ea024', 'request_id': 'req-e4e97693-1ee1-43ba-83ff-bd01088350f7'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:11.950808 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-e4e97693-1ee1-43ba-83ff-bd01088350f7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '105497a27f944bfdb9e844ffb00ea024', 'request_id': 'req-e4e97693-1ee1-43ba-83ff-bd01088350f7'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:11.961966 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-e4e97693-1ee1-43ba-83ff-bd01088350f7 admin admin] The token cache is being invalidate because project 105497a27f944bfdb9e844ffb00ea024 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:34:11.962258 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-e4e97693-1ee1-43ba-83ff-bd01088350f7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:11.962920 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-e4e97693-1ee1-43ba-83ff-bd01088350f7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:11.964492 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1172/2332] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:34:11 2022] DELETE /identity/v3/projects/105497a27f944bfdb9e844ffb00ea024 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:34:11.968513 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4f17ceb-2322-4c39-8598-e31bed555af6 service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:11.978870 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4f17ceb-2322-4c39-8598-e31bed555af6 service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:12.041504 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68269da8-1836-4b6f-95db-92f1c4196c2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:12.042138 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68269da8-1836-4b6f-95db-92f1c4196c2b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:12.042415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68269da8-1836-4b6f-95db-92f1c4196c2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:12.042687 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68269da8-1836-4b6f-95db-92f1c4196c2b admin admin] PATH_INFO: `/v3/users/18b360d4555d4f3c9fd64e35ee90f9c5` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:12.043807 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68269da8-1836-4b6f-95db-92f1c4196c2b admin admin] RBAC: Authorizing `identity:delete_user(user_id=18b360d4555d4f3c9fd64e35ee90f9c5)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:12.045784 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68269da8-1836-4b6f-95db-92f1c4196c2b admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:12.056326 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:34:12.093804 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-68269da8-1836-4b6f-95db-92f1c4196c2b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '18b360d4555d4f3c9fd64e35ee90f9c5'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:12.102568 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-68269da8-1836-4b6f-95db-92f1c4196c2b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '18b360d4555d4f3c9fd64e35ee90f9c5'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:12.103227 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-68269da8-1836-4b6f-95db-92f1c4196c2b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '18b360d4555d4f3c9fd64e35ee90f9c5'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:12.115604 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-68269da8-1836-4b6f-95db-92f1c4196c2b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '18b360d4555d4f3c9fd64e35ee90f9c5'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:12.122667 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-68269da8-1836-4b6f-95db-92f1c4196c2b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '18b360d4555d4f3c9fd64e35ee90f9c5'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:12.128688 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1161/2333] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:34:11 2022] DELETE /identity/v3/users/18b360d4555d4f3c9fd64e35ee90f9c5 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:34:12.401043 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e97693-1ee1-43ba-83ff-bd01088350f7 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:12.402609 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-e4e97693-1ee1-43ba-83ff-bd01088350f7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:12.415918 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e97693-1ee1-43ba-83ff-bd01088350f7 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:12.474730 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ddb69c-9ada-40b4-ac8e-8680487c51b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:12.475359 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ddb69c-9ada-40b4-ac8e-8680487c51b6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:12.475633 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ddb69c-9ada-40b4-ac8e-8680487c51b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:12.475899 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ddb69c-9ada-40b4-ac8e-8680487c51b6 admin admin] PATH_INFO: `/v3/projects/81f90a474e434ed6ab3ed1b03556415e` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:12.477165 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83ddb69c-9ada-40b4-ac8e-8680487c51b6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=81f90a474e434ed6ab3ed1b03556415e)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:12.479050 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83ddb69c-9ada-40b4-ac8e-8680487c51b6 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:12.527820 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-83ddb69c-9ada-40b4-ac8e-8680487c51b6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '81f90a474e434ed6ab3ed1b03556415e', 'request_id': 'req-83ddb69c-9ada-40b4-ac8e-8680487c51b6'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:12.535434 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-83ddb69c-9ada-40b4-ac8e-8680487c51b6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '81f90a474e434ed6ab3ed1b03556415e', 'request_id': 'req-83ddb69c-9ada-40b4-ac8e-8680487c51b6'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:12.542319 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-83ddb69c-9ada-40b4-ac8e-8680487c51b6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '81f90a474e434ed6ab3ed1b03556415e', 'request_id': 'req-83ddb69c-9ada-40b4-ac8e-8680487c51b6'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:12.554419 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-83ddb69c-9ada-40b4-ac8e-8680487c51b6 admin admin] The token cache is being invalidate because project 81f90a474e434ed6ab3ed1b03556415e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:34:12.554713 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-83ddb69c-9ada-40b4-ac8e-8680487c51b6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:12.555367 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-83ddb69c-9ada-40b4-ac8e-8680487c51b6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:12.556869 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1173/2334] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:34:12 2022] DELETE /identity/v3/projects/81f90a474e434ed6ab3ed1b03556415e => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:34:12.561698 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49ee2eb-f874-4cdd-8096-52df840207d2 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:12.561984 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49ee2eb-f874-4cdd-8096-52df840207d2 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:12.562257 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49ee2eb-f874-4cdd-8096-52df840207d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:12.583191 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-d49ee2eb-f874-4cdd-8096-52df840207d2 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:34:12.584439 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d49ee2eb-f874-4cdd-8096-52df840207d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:12.651738 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1162/2335] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:34:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:34:12.660310 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8c1beb-88da-4d8a-b8e7-95e490ddfe2a None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:12.661186 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8c1beb-88da-4d8a-b8e7-95e490ddfe2a None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:12.662147 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8c1beb-88da-4d8a-b8e7-95e490ddfe2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:12.683585 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-af8c1beb-88da-4d8a-b8e7-95e490ddfe2a None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:34:12.684801 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-af8c1beb-88da-4d8a-b8e7-95e490ddfe2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:12.694015 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1174/2336] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:34:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:34:12.698578 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49ee2eb-f874-4cdd-8096-52df840207d2 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:12.713597 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49ee2eb-f874-4cdd-8096-52df840207d2 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:12.718151 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d287f0b-e70e-4837-9b38-1120e6d1c8e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:12.718752 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d287f0b-e70e-4837-9b38-1120e6d1c8e5 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:12.719017 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d287f0b-e70e-4837-9b38-1120e6d1c8e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:12.719303 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d287f0b-e70e-4837-9b38-1120e6d1c8e5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:12.719991 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d287f0b-e70e-4837-9b38-1120e6d1c8e5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:34:12.720295 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d287f0b-e70e-4837-9b38-1120e6d1c8e5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:12.721917 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d287f0b-e70e-4837-9b38-1120e6d1c8e5 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:12.734472 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1163/2337] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:34:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:34:12.739391 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af8c1beb-88da-4d8a-b8e7-95e490ddfe2a None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:12.748603 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af8c1beb-88da-4d8a-b8e7-95e490ddfe2a None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:12.753318 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b095a728-6cd1-453e-a617-f263b14227be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:12.753912 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b095a728-6cd1-453e-a617-f263b14227be admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:12.754192 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b095a728-6cd1-453e-a617-f263b14227be admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:12.754505 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b095a728-6cd1-453e-a617-f263b14227be admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:12.755274 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b095a728-6cd1-453e-a617-f263b14227be admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:12.757198 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b095a728-6cd1-453e-a617-f263b14227be admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:12.778413 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:34:12.789969 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1175/2338] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:34:12 2022] POST /identity/v3/projects => generated 367 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:34:12.790926 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d287f0b-e70e-4837-9b38-1120e6d1c8e5 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:12.801290 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d287f0b-e70e-4837-9b38-1120e6d1c8e5 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:12.858704 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dd32c8-a9f0-427f-b1a6-ec440338589d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:12.859292 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dd32c8-a9f0-427f-b1a6-ec440338589d admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:12.859570 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dd32c8-a9f0-427f-b1a6-ec440338589d admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:12.859842 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dd32c8-a9f0-427f-b1a6-ec440338589d admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:12.860467 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6dd32c8-a9f0-427f-b1a6-ec440338589d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:12.862409 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6dd32c8-a9f0-427f-b1a6-ec440338589d admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:12.883026 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1164/2339] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:34:12 2022] POST /identity/v3/users => generated 284 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:34:12.887337 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b095a728-6cd1-453e-a617-f263b14227be admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:12.898417 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b095a728-6cd1-453e-a617-f263b14227be admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:12.903035 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123de5a9-349c-4513-a0c5-19550128faa1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:12.903630 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123de5a9-349c-4513-a0c5-19550128faa1 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:12.903911 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123de5a9-349c-4513-a0c5-19550128faa1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:12.904184 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123de5a9-349c-4513-a0c5-19550128faa1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:12.904821 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-123de5a9-349c-4513-a0c5-19550128faa1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:12.906448 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-123de5a9-349c-4513-a0c5-19550128faa1 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:12.917740 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1176/2340] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:34:12 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:34:12.921617 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dd32c8-a9f0-427f-b1a6-ec440338589d admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:12.928296 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:DELETE=10,keystone:SELECT=86,keystone:INSERT=4 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:34:12.932100 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dd32c8-a9f0-427f-b1a6-ec440338589d admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:12.938174 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cdbb937-82f5-4088-8d56-fd93dc4d1e4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:12.938996 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdbb937-82f5-4088-8d56-fd93dc4d1e4f admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:12.939286 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdbb937-82f5-4088-8d56-fd93dc4d1e4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:12.939666 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdbb937-82f5-4088-8d56-fd93dc4d1e4f admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:12.940484 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cdbb937-82f5-4088-8d56-fd93dc4d1e4f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:12.942319 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cdbb937-82f5-4088-8d56-fd93dc4d1e4f admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:12.952979 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1165/2341] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:34:12 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:34:12.957905 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123de5a9-349c-4513-a0c5-19550128faa1 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:12.967717 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123de5a9-349c-4513-a0c5-19550128faa1 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:12.972638 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c21674-061b-438d-983a-688e4f3c3696 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:12.973463 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c21674-061b-438d-983a-688e4f3c3696 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:12.973828 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c21674-061b-438d-983a-688e4f3c3696 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:12.974101 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c21674-061b-438d-983a-688e4f3c3696 admin admin] PATH_INFO: `/v3/projects/8bb1b75cc5d04740921563d834113140/users/9ffe6027b82647c78bab1e0288980a5a/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:12.988524 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c21674-061b-438d-983a-688e4f3c3696 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8bb1b75cc5d04740921563d834113140, user_id=9ffe6027b82647c78bab1e0288980a5a, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:12.991913 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c21674-061b-438d-983a-688e4f3c3696 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:13.004987 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1177/2342] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:34:12 2022] PUT /identity/v3/projects/8bb1b75cc5d04740921563d834113140/users/9ffe6027b82647c78bab1e0288980a5a/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:34:13.015134 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ae9075-e524-4f35-9ac9-57983521cd82 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:13.015416 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ae9075-e524-4f35-9ac9-57983521cd82 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:13.015682 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ae9075-e524-4f35-9ac9-57983521cd82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:13.042556 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-77ae9075-e524-4f35-9ac9-57983521cd82 None None] MFA Rules not processed for user `9ffe6027b82647c78bab1e0288980a5a`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:34:13.043790 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-77ae9075-e524-4f35-9ac9-57983521cd82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:13.111181 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1166/2343] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:34:13 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:34:13.116316 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c21674-061b-438d-983a-688e4f3c3696 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:13.179127 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c21674-061b-438d-983a-688e4f3c3696 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:13.183748 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b615d4-3817-422f-811b-907e740f0fae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:13.184346 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b615d4-3817-422f-811b-907e740f0fae admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:13.184624 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b615d4-3817-422f-811b-907e740f0fae admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:13.184892 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b615d4-3817-422f-811b-907e740f0fae admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:13.185553 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4b615d4-3817-422f-811b-907e740f0fae admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:13.187542 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4b615d4-3817-422f-811b-907e740f0fae admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:13.213293 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1178/2344] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:34:13 2022] POST /identity/v3/projects => generated 367 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:34:13.217348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ae9075-e524-4f35-9ac9-57983521cd82 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:13.228379 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ae9075-e524-4f35-9ac9-57983521cd82 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:13.282322 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d724b05-4af1-4b76-afe8-b1c76d205bd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:13.282908 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d724b05-4af1-4b76-afe8-b1c76d205bd9 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:13.283181 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d724b05-4af1-4b76-afe8-b1c76d205bd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:13.283449 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d724b05-4af1-4b76-afe8-b1c76d205bd9 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:13.284069 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d724b05-4af1-4b76-afe8-b1c76d205bd9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:13.286490 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d724b05-4af1-4b76-afe8-b1c76d205bd9 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:13.307990 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1167/2345] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:34:13 2022] POST /identity/v3/users => generated 290 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:34:13.312180 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b615d4-3817-422f-811b-907e740f0fae admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:13.322448 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b615d4-3817-422f-811b-907e740f0fae admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:13.327100 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db1d396-a5d7-484e-8f55-1da88b3350fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:13.327669 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db1d396-a5d7-484e-8f55-1da88b3350fd admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:13.327948 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db1d396-a5d7-484e-8f55-1da88b3350fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:13.328219 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db1d396-a5d7-484e-8f55-1da88b3350fd admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:13.328852 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4db1d396-a5d7-484e-8f55-1da88b3350fd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:13.330489 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4db1d396-a5d7-484e-8f55-1da88b3350fd admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:13.340830 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1179/2346] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:34:13 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:34:13.344785 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d724b05-4af1-4b76-afe8-b1c76d205bd9 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:13.355862 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d724b05-4af1-4b76-afe8-b1c76d205bd9 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:13.360479 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d7d2c7-ee2b-41bc-9ee6-7c844c6283a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:13.361154 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d7d2c7-ee2b-41bc-9ee6-7c844c6283a3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:13.361433 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d7d2c7-ee2b-41bc-9ee6-7c844c6283a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:13.361702 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d7d2c7-ee2b-41bc-9ee6-7c844c6283a3 admin admin] PATH_INFO: `/v3/projects/74ff87b56396429592c365c6649cf72e/users/0e303ae11a294dffaffbb719511dac5b/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:13.377736 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d7d2c7-ee2b-41bc-9ee6-7c844c6283a3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=74ff87b56396429592c365c6649cf72e, user_id=0e303ae11a294dffaffbb719511dac5b, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:13.380922 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d7d2c7-ee2b-41bc-9ee6-7c844c6283a3 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:13.393425 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1168/2347] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:34:13 2022] PUT /identity/v3/projects/74ff87b56396429592c365c6649cf72e/users/0e303ae11a294dffaffbb719511dac5b/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:34:13.397889 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db1d396-a5d7-484e-8f55-1da88b3350fd admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:13.407557 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db1d396-a5d7-484e-8f55-1da88b3350fd admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:13.464065 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3baa33c-d330-4f60-9aa4-b4e13f6592f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:13.464897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3baa33c-d330-4f60-9aa4-b4e13f6592f5 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:13.465214 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3baa33c-d330-4f60-9aa4-b4e13f6592f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:13.465489 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3baa33c-d330-4f60-9aa4-b4e13f6592f5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:13.466127 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3baa33c-d330-4f60-9aa4-b4e13f6592f5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:13.467712 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3baa33c-d330-4f60-9aa4-b4e13f6592f5 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:13.477763 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1180/2348] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:34:13 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:34:13.481904 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d7d2c7-ee2b-41bc-9ee6-7c844c6283a3 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:13.493205 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d7d2c7-ee2b-41bc-9ee6-7c844c6283a3 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:13.497814 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0939b995-87bd-4747-9955-d0de44b5f59e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:13.498455 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0939b995-87bd-4747-9955-d0de44b5f59e admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:13.498729 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0939b995-87bd-4747-9955-d0de44b5f59e admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:13.498996 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0939b995-87bd-4747-9955-d0de44b5f59e admin admin] PATH_INFO: `/v3/projects/74ff87b56396429592c365c6649cf72e/users/0e303ae11a294dffaffbb719511dac5b/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:13.500864 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0939b995-87bd-4747-9955-d0de44b5f59e admin admin] RBAC: Authorizing `identity:create_grant(project_id=74ff87b56396429592c365c6649cf72e, user_id=0e303ae11a294dffaffbb719511dac5b, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:13.503530 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0939b995-87bd-4747-9955-d0de44b5f59e admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:13.513989 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1169/2349] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:34:13 2022] PUT /identity/v3/projects/74ff87b56396429592c365c6649cf72e/users/0e303ae11a294dffaffbb719511dac5b/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:34:13.525995 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c365fd6-4ea2-4008-b1c1-0fd37d58dd8d None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:13.526278 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c365fd6-4ea2-4008-b1c1-0fd37d58dd8d None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:13.526552 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c365fd6-4ea2-4008-b1c1-0fd37d58dd8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:13.554754 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-2c365fd6-4ea2-4008-b1c1-0fd37d58dd8d None None] MFA Rules not processed for user `0e303ae11a294dffaffbb719511dac5b`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:34:13.555998 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-2c365fd6-4ea2-4008-b1c1-0fd37d58dd8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:13.579484 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:34:13.624694 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1181/2350] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:34:13 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:34:13.636464 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0939b995-87bd-4747-9955-d0de44b5f59e admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:13.638174 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0939b995-87bd-4747-9955-d0de44b5f59e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:13.678949 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:34:13.697026 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0939b995-87bd-4747-9955-d0de44b5f59e admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:13.700675 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f9dd8e-f532-4030-9d91-fe8eda2af87b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:13.701298 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f9dd8e-f532-4030-9d91-fe8eda2af87b service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:13.701575 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f9dd8e-f532-4030-9d91-fe8eda2af87b service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:13.701843 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f9dd8e-f532-4030-9d91-fe8eda2af87b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:13.711044 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f9dd8e-f532-4030-9d91-fe8eda2af87b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:13.712501 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f9dd8e-f532-4030-9d91-fe8eda2af87b service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:13.721850 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1170/2351] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:34:13 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:34:13.810212 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c365fd6-4ea2-4008-b1c1-0fd37d58dd8d None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:13.820423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c365fd6-4ea2-4008-b1c1-0fd37d58dd8d None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:13.824209 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6552c5c6-dcc8-4961-9568-d8dfa9f3cae0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:13.824960 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6552c5c6-dcc8-4961-9568-d8dfa9f3cae0 service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:13.825316 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6552c5c6-dcc8-4961-9568-d8dfa9f3cae0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:13.825654 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6552c5c6-dcc8-4961-9568-d8dfa9f3cae0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:13.886482 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6552c5c6-dcc8-4961-9568-d8dfa9f3cae0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:13.887923 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6552c5c6-dcc8-4961-9568-d8dfa9f3cae0 service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:13.897979 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1182/2352] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:34:13 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:34:18.695361 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f9dd8e-f532-4030-9d91-fe8eda2af87b service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:18.710562 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6552c5c6-dcc8-4961-9568-d8dfa9f3cae0 service cinder] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:18.712000 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-6552c5c6-dcc8-4961-9568-d8dfa9f3cae0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:18.761707 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f9dd8e-f532-4030-9d91-fe8eda2af87b service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:18.766188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d32b3d-84dd-4192-9003-ff64c6dca840 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:18.766810 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d32b3d-84dd-4192-9003-ff64c6dca840 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:18.767097 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d32b3d-84dd-4192-9003-ff64c6dca840 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:18.767427 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d32b3d-84dd-4192-9003-ff64c6dca840 admin admin] PATH_INFO: `/v3/users/9ffe6027b82647c78bab1e0288980a5a` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:18.768456 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15d32b3d-84dd-4192-9003-ff64c6dca840 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9ffe6027b82647c78bab1e0288980a5a)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:18.771497 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15d32b3d-84dd-4192-9003-ff64c6dca840 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:18.772747 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6552c5c6-dcc8-4961-9568-d8dfa9f3cae0 service cinder] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:18.776283 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8284c0a-96db-4c99-a9b8-59b54071bb49 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:18.776879 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8284c0a-96db-4c99-a9b8-59b54071bb49 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:18.777208 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8284c0a-96db-4c99-a9b8-59b54071bb49 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:18.777474 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8284c0a-96db-4c99-a9b8-59b54071bb49 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:18.779194 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-b8284c0a-96db-4c99-a9b8-59b54071bb49 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:18.789448 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8284c0a-96db-4c99-a9b8-59b54071bb49 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:18.790906 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8284c0a-96db-4c99-a9b8-59b54071bb49 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:18.808406 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1183/2353] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:34:18 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 27 01:34:18.825177 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-15d32b3d-84dd-4192-9003-ff64c6dca840 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9ffe6027b82647c78bab1e0288980a5a'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:18.833101 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-15d32b3d-84dd-4192-9003-ff64c6dca840 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9ffe6027b82647c78bab1e0288980a5a'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:18.833620 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-15d32b3d-84dd-4192-9003-ff64c6dca840 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9ffe6027b82647c78bab1e0288980a5a'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:18.846652 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-15d32b3d-84dd-4192-9003-ff64c6dca840 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9ffe6027b82647c78bab1e0288980a5a'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:18.854906 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-15d32b3d-84dd-4192-9003-ff64c6dca840 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9ffe6027b82647c78bab1e0288980a5a'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:18.860862 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1171/2354] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:34:18 2022] DELETE /identity/v3/users/9ffe6027b82647c78bab1e0288980a5a => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:34:18.866056 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cb41b0-36f6-43e9-87f9-b9156203784d None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:18.866332 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cb41b0-36f6-43e9-87f9-b9156203784d None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:18.866599 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cb41b0-36f6-43e9-87f9-b9156203784d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:18.886394 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-40cb41b0-36f6-43e9-87f9-b9156203784d None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:34:18.887585 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-40cb41b0-36f6-43e9-87f9-b9156203784d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:18.949670 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1184/2355] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:34:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:34:18.959819 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d32b3d-84dd-4192-9003-ff64c6dca840 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:18.961279 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-15d32b3d-84dd-4192-9003-ff64c6dca840 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:19.024436 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d32b3d-84dd-4192-9003-ff64c6dca840 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:19.028074 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a6bfc1-bc21-4ace-a80b-2c1ed0283070 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:19.028687 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a6bfc1-bc21-4ace-a80b-2c1ed0283070 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:19.028965 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a6bfc1-bc21-4ace-a80b-2c1ed0283070 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:19.029279 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a6bfc1-bc21-4ace-a80b-2c1ed0283070 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:19.040472 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a6bfc1-bc21-4ace-a80b-2c1ed0283070 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:19.041982 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a6bfc1-bc21-4ace-a80b-2c1ed0283070 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:19.048383 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:34:19.056789 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1172/2356] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:34:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:34:19.330623 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cb41b0-36f6-43e9-87f9-b9156203784d None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:19.332838 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-40cb41b0-36f6-43e9-87f9-b9156203784d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:19.340730 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:34:19.347549 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cb41b0-36f6-43e9-87f9-b9156203784d None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:19.352956 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1305a483-52f3-4072-bd06-cbd93b1218b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:19.353790 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1305a483-52f3-4072-bd06-cbd93b1218b5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:19.354198 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1305a483-52f3-4072-bd06-cbd93b1218b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:19.354340 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1305a483-52f3-4072-bd06-cbd93b1218b5 admin admin] PATH_INFO: `/v3/projects/8bb1b75cc5d04740921563d834113140` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:19.355654 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1305a483-52f3-4072-bd06-cbd93b1218b5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8bb1b75cc5d04740921563d834113140)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:19.358225 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1305a483-52f3-4072-bd06-cbd93b1218b5 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:19.408273 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-1305a483-52f3-4072-bd06-cbd93b1218b5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8bb1b75cc5d04740921563d834113140', 'request_id': 'req-1305a483-52f3-4072-bd06-cbd93b1218b5'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:19.415844 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-1305a483-52f3-4072-bd06-cbd93b1218b5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8bb1b75cc5d04740921563d834113140', 'request_id': 'req-1305a483-52f3-4072-bd06-cbd93b1218b5'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:19.422403 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-1305a483-52f3-4072-bd06-cbd93b1218b5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8bb1b75cc5d04740921563d834113140', 'request_id': 'req-1305a483-52f3-4072-bd06-cbd93b1218b5'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:19.432590 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-1305a483-52f3-4072-bd06-cbd93b1218b5 admin admin] The token cache is being invalidate because project 8bb1b75cc5d04740921563d834113140 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:34:19.432894 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-1305a483-52f3-4072-bd06-cbd93b1218b5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:19.433532 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-1305a483-52f3-4072-bd06-cbd93b1218b5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:19.435004 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1185/2357] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:34:19 2022] DELETE /identity/v3/projects/8bb1b75cc5d04740921563d834113140 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:34:19.439167 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a6bfc1-bc21-4ace-a80b-2c1ed0283070 service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:19.449558 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a6bfc1-bc21-4ace-a80b-2c1ed0283070 service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:19.507728 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714ca542-ce71-42e9-984b-7b46a8a6a7f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:19.508357 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714ca542-ce71-42e9-984b-7b46a8a6a7f8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:19.508635 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714ca542-ce71-42e9-984b-7b46a8a6a7f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:19.508908 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714ca542-ce71-42e9-984b-7b46a8a6a7f8 admin admin] PATH_INFO: `/v3/users/0e303ae11a294dffaffbb719511dac5b` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:19.510064 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-714ca542-ce71-42e9-984b-7b46a8a6a7f8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0e303ae11a294dffaffbb719511dac5b)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:19.511985 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-714ca542-ce71-42e9-984b-7b46a8a6a7f8 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:19.556696 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-714ca542-ce71-42e9-984b-7b46a8a6a7f8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0e303ae11a294dffaffbb719511dac5b'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:19.563456 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-714ca542-ce71-42e9-984b-7b46a8a6a7f8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0e303ae11a294dffaffbb719511dac5b'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:19.564081 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-714ca542-ce71-42e9-984b-7b46a8a6a7f8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0e303ae11a294dffaffbb719511dac5b'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:19.576556 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-714ca542-ce71-42e9-984b-7b46a8a6a7f8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0e303ae11a294dffaffbb719511dac5b'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:19.583685 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-714ca542-ce71-42e9-984b-7b46a8a6a7f8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0e303ae11a294dffaffbb719511dac5b'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:19.589391 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1173/2358] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:34:19 2022] DELETE /identity/v3/users/0e303ae11a294dffaffbb719511dac5b => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:34:19.932605 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1305a483-52f3-4072-bd06-cbd93b1218b5 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:19.934322 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-1305a483-52f3-4072-bd06-cbd93b1218b5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:19.948311 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1305a483-52f3-4072-bd06-cbd93b1218b5 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:20.008255 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f88d776-906d-44e3-9170-69dced075fc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:20.008876 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f88d776-906d-44e3-9170-69dced075fc3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:20.009180 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f88d776-906d-44e3-9170-69dced075fc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:20.009443 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f88d776-906d-44e3-9170-69dced075fc3 admin admin] PATH_INFO: `/v3/projects/74ff87b56396429592c365c6649cf72e` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:20.010533 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f88d776-906d-44e3-9170-69dced075fc3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=74ff87b56396429592c365c6649cf72e)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:20.012415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f88d776-906d-44e3-9170-69dced075fc3 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:20.058775 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-3f88d776-906d-44e3-9170-69dced075fc3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '74ff87b56396429592c365c6649cf72e', 'request_id': 'req-3f88d776-906d-44e3-9170-69dced075fc3'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:20.065917 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-3f88d776-906d-44e3-9170-69dced075fc3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '74ff87b56396429592c365c6649cf72e', 'request_id': 'req-3f88d776-906d-44e3-9170-69dced075fc3'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:20.073189 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-3f88d776-906d-44e3-9170-69dced075fc3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '74ff87b56396429592c365c6649cf72e', 'request_id': 'req-3f88d776-906d-44e3-9170-69dced075fc3'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:20.083356 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-3f88d776-906d-44e3-9170-69dced075fc3 admin admin] The token cache is being invalidate because project 74ff87b56396429592c365c6649cf72e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:34:20.083577 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-3f88d776-906d-44e3-9170-69dced075fc3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:20.084200 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-3f88d776-906d-44e3-9170-69dced075fc3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:20.085690 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1186/2359] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:34:19 2022] DELETE /identity/v3/projects/74ff87b56396429592c365c6649cf72e => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:34:20.090432 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df853710-1f44-489e-a969-32443958ac9d None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:20.090936 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df853710-1f44-489e-a969-32443958ac9d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:20.091035 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df853710-1f44-489e-a969-32443958ac9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:20.110668 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-df853710-1f44-489e-a969-32443958ac9d None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:34:20.111858 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-df853710-1f44-489e-a969-32443958ac9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:20.169348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1174/2360] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:34:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:34:20.177415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f18799-f42e-4e56-8c85-7066f4554d0e None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:20.177718 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f18799-f42e-4e56-8c85-7066f4554d0e None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:20.177992 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f18799-f42e-4e56-8c85-7066f4554d0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:20.198009 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-01f18799-f42e-4e56-8c85-7066f4554d0e None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:34:20.199211 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-01f18799-f42e-4e56-8c85-7066f4554d0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:20.207717 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1187/2361] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:34:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:34:20.212269 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df853710-1f44-489e-a969-32443958ac9d None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:20.221875 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df853710-1f44-489e-a969-32443958ac9d None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:20.226898 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca86818-37ae-4be9-abc7-bd3c23ada6a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:20.227479 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca86818-37ae-4be9-abc7-bd3c23ada6a9 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:20.227752 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca86818-37ae-4be9-abc7-bd3c23ada6a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:20.228017 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca86818-37ae-4be9-abc7-bd3c23ada6a9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:20.228646 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ca86818-37ae-4be9-abc7-bd3c23ada6a9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:34:20.228938 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ca86818-37ae-4be9-abc7-bd3c23ada6a9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:20.230559 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ca86818-37ae-4be9-abc7-bd3c23ada6a9 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:20.242067 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1175/2362] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:34:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:34:20.246268 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f18799-f42e-4e56-8c85-7066f4554d0e None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:20.255928 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f18799-f42e-4e56-8c85-7066f4554d0e None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:20.260578 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eda559e-5f43-4802-a844-725608021cf6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:20.261209 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eda559e-5f43-4802-a844-725608021cf6 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:20.261487 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eda559e-5f43-4802-a844-725608021cf6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:20.261755 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eda559e-5f43-4802-a844-725608021cf6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:20.262378 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eda559e-5f43-4802-a844-725608021cf6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:20.264265 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eda559e-5f43-4802-a844-725608021cf6 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:20.269531 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:34:20.293707 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1188/2363] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:34:20 2022] POST /identity/v3/projects => generated 371 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:34:20.298193 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca86818-37ae-4be9-abc7-bd3c23ada6a9 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:20.308670 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca86818-37ae-4be9-abc7-bd3c23ada6a9 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:20.312155 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:34:20.367066 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea169a9-8367-4935-8b12-9bca69a29bcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:20.367589 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea169a9-8367-4935-8b12-9bca69a29bcd admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:20.367862 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea169a9-8367-4935-8b12-9bca69a29bcd admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:20.368128 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea169a9-8367-4935-8b12-9bca69a29bcd admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:20.368743 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea169a9-8367-4935-8b12-9bca69a29bcd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:20.370671 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea169a9-8367-4935-8b12-9bca69a29bcd admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:20.391495 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1176/2364] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:34:20 2022] POST /identity/v3/users => generated 286 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:34:20.395639 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eda559e-5f43-4802-a844-725608021cf6 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:20.406541 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eda559e-5f43-4802-a844-725608021cf6 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:20.411094 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece23b3a-d8d5-4900-884d-a440dda86de8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:20.411678 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece23b3a-d8d5-4900-884d-a440dda86de8 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:20.411951 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece23b3a-d8d5-4900-884d-a440dda86de8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:20.412216 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece23b3a-d8d5-4900-884d-a440dda86de8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:20.412839 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece23b3a-d8d5-4900-884d-a440dda86de8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:20.414458 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece23b3a-d8d5-4900-884d-a440dda86de8 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:20.424235 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1189/2365] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:34:20 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:34:20.428807 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea169a9-8367-4935-8b12-9bca69a29bcd admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:20.439705 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea169a9-8367-4935-8b12-9bca69a29bcd admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:20.445260 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a168df-4895-40af-9f4f-13c7dbfe5c48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:20.445856 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a168df-4895-40af-9f4f-13c7dbfe5c48 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:20.446133 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a168df-4895-40af-9f4f-13c7dbfe5c48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:20.446403 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a168df-4895-40af-9f4f-13c7dbfe5c48 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:20.447043 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a168df-4895-40af-9f4f-13c7dbfe5c48 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:20.448816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a168df-4895-40af-9f4f-13c7dbfe5c48 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:20.460609 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1177/2366] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:34:20 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:34:20.464737 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece23b3a-d8d5-4900-884d-a440dda86de8 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:20.475830 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece23b3a-d8d5-4900-884d-a440dda86de8 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:20.482203 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6079698e-8e4e-485c-bc5b-1c7917de1409 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:20.482203 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6079698e-8e4e-485c-bc5b-1c7917de1409 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:20.482203 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6079698e-8e4e-485c-bc5b-1c7917de1409 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:20.482616 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6079698e-8e4e-485c-bc5b-1c7917de1409 admin admin] PATH_INFO: `/v3/projects/89bed199e23b48dfa4bab7846705d6f3/users/a52f4b8af9de40d79e30dd0c710445b8/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:20.500721 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6079698e-8e4e-485c-bc5b-1c7917de1409 admin admin] RBAC: Authorizing `identity:create_grant(project_id=89bed199e23b48dfa4bab7846705d6f3, user_id=a52f4b8af9de40d79e30dd0c710445b8, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:20.503450 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6079698e-8e4e-485c-bc5b-1c7917de1409 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:20.516874 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1190/2367] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:34:20 2022] PUT /identity/v3/projects/89bed199e23b48dfa4bab7846705d6f3/users/a52f4b8af9de40d79e30dd0c710445b8/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:34:20.528564 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdae4d36-fda5-42e0-8aef-2d8d53d77e20 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:20.528845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdae4d36-fda5-42e0-8aef-2d8d53d77e20 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:20.529152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdae4d36-fda5-42e0-8aef-2d8d53d77e20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:20.557824 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-cdae4d36-fda5-42e0-8aef-2d8d53d77e20 None None] MFA Rules not processed for user `a52f4b8af9de40d79e30dd0c710445b8`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:34:20.559019 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-cdae4d36-fda5-42e0-8aef-2d8d53d77e20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:20.622505 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1178/2368] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:34:20 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:34:20.633841 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6079698e-8e4e-485c-bc5b-1c7917de1409 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:20.635522 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-6079698e-8e4e-485c-bc5b-1c7917de1409 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:20.691184 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6079698e-8e4e-485c-bc5b-1c7917de1409 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:20.695349 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491acb88-f607-4b28-972a-3956bdfc0254 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:20.695948 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491acb88-f607-4b28-972a-3956bdfc0254 service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:20.696224 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491acb88-f607-4b28-972a-3956bdfc0254 service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:20.696494 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491acb88-f607-4b28-972a-3956bdfc0254 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:20.704806 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-491acb88-f607-4b28-972a-3956bdfc0254 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:20.706267 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-491acb88-f607-4b28-972a-3956bdfc0254 service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:20.715905 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1191/2369] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:34:20 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:34:20.737314 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdae4d36-fda5-42e0-8aef-2d8d53d77e20 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:20.798689 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdae4d36-fda5-42e0-8aef-2d8d53d77e20 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:20.803510 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038bba36-fd99-4c8a-bb84-2da2d93e04aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:20.804141 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038bba36-fd99-4c8a-bb84-2da2d93e04aa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:20.804419 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038bba36-fd99-4c8a-bb84-2da2d93e04aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:20.804690 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038bba36-fd99-4c8a-bb84-2da2d93e04aa admin admin] PATH_INFO: `/v3/users/a52f4b8af9de40d79e30dd0c710445b8` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:20.805867 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038bba36-fd99-4c8a-bb84-2da2d93e04aa admin admin] RBAC: Authorizing `identity:delete_user(user_id=a52f4b8af9de40d79e30dd0c710445b8)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:20.807789 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038bba36-fd99-4c8a-bb84-2da2d93e04aa admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:20.825654 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=96,keystone:INSERT=3,keystone:DELETE=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:34:20.856863 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-038bba36-fd99-4c8a-bb84-2da2d93e04aa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a52f4b8af9de40d79e30dd0c710445b8'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:20.865809 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-038bba36-fd99-4c8a-bb84-2da2d93e04aa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a52f4b8af9de40d79e30dd0c710445b8'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:20.866545 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-038bba36-fd99-4c8a-bb84-2da2d93e04aa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a52f4b8af9de40d79e30dd0c710445b8'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:20.880458 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-038bba36-fd99-4c8a-bb84-2da2d93e04aa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a52f4b8af9de40d79e30dd0c710445b8'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:20.887398 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-038bba36-fd99-4c8a-bb84-2da2d93e04aa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a52f4b8af9de40d79e30dd0c710445b8'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:20.893594 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1179/2370] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:34:20 2022] DELETE /identity/v3/users/a52f4b8af9de40d79e30dd0c710445b8 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:34:20.898684 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcda6fd9-78d1-40d6-a06b-64acf6f08ad5 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:20.898964 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcda6fd9-78d1-40d6-a06b-64acf6f08ad5 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:20.899258 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcda6fd9-78d1-40d6-a06b-64acf6f08ad5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:20.918714 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-fcda6fd9-78d1-40d6-a06b-64acf6f08ad5 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:34:20.919953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-fcda6fd9-78d1-40d6-a06b-64acf6f08ad5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:20.983692 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1192/2371] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:34:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:34:20.993541 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038bba36-fd99-4c8a-bb84-2da2d93e04aa admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:20.994952 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-038bba36-fd99-4c8a-bb84-2da2d93e04aa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:21.055317 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038bba36-fd99-4c8a-bb84-2da2d93e04aa admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:21.059343 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f15670d-cd95-47f9-9d7d-d5c23627b89b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:21.059700 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f15670d-cd95-47f9-9d7d-d5c23627b89b service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:21.059976 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f15670d-cd95-47f9-9d7d-d5c23627b89b service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:21.060242 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f15670d-cd95-47f9-9d7d-d5c23627b89b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:21.071235 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f15670d-cd95-47f9-9d7d-d5c23627b89b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:21.072375 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f15670d-cd95-47f9-9d7d-d5c23627b89b service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:21.081896 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1180/2372] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:34:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:34:21.369717 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcda6fd9-78d1-40d6-a06b-64acf6f08ad5 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:21.371519 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-fcda6fd9-78d1-40d6-a06b-64acf6f08ad5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:21.384926 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcda6fd9-78d1-40d6-a06b-64acf6f08ad5 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:21.390545 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057c6afc-bc79-4590-8dae-dcf32bac95fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:21.391154 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057c6afc-bc79-4590-8dae-dcf32bac95fe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:21.391442 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057c6afc-bc79-4590-8dae-dcf32bac95fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:21.391743 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057c6afc-bc79-4590-8dae-dcf32bac95fe admin admin] PATH_INFO: `/v3/projects/89bed199e23b48dfa4bab7846705d6f3` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:21.393257 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-057c6afc-bc79-4590-8dae-dcf32bac95fe admin admin] RBAC: Authorizing `identity:delete_project(project_id=89bed199e23b48dfa4bab7846705d6f3)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:21.395652 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-057c6afc-bc79-4590-8dae-dcf32bac95fe admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:21.450194 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-057c6afc-bc79-4590-8dae-dcf32bac95fe admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '89bed199e23b48dfa4bab7846705d6f3', 'request_id': 'req-057c6afc-bc79-4590-8dae-dcf32bac95fe'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:21.452059 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=94,keystone:INSERT=2,keystone:DELETE=4 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:34:21.470068 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-057c6afc-bc79-4590-8dae-dcf32bac95fe admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '89bed199e23b48dfa4bab7846705d6f3', 'request_id': 'req-057c6afc-bc79-4590-8dae-dcf32bac95fe'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:21.494141 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-057c6afc-bc79-4590-8dae-dcf32bac95fe admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '89bed199e23b48dfa4bab7846705d6f3', 'request_id': 'req-057c6afc-bc79-4590-8dae-dcf32bac95fe'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:21.505819 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-057c6afc-bc79-4590-8dae-dcf32bac95fe admin admin] The token cache is being invalidate because project 89bed199e23b48dfa4bab7846705d6f3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:34:21.506199 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-057c6afc-bc79-4590-8dae-dcf32bac95fe admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:21.506908 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-057c6afc-bc79-4590-8dae-dcf32bac95fe admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:21.508627 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1193/2373] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:34:21 2022] DELETE /identity/v3/projects/89bed199e23b48dfa4bab7846705d6f3 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:34:21.513451 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b233ef2-c118-49b5-b87d-85206559248f None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:21.513922 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b233ef2-c118-49b5-b87d-85206559248f None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:21.514031 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b233ef2-c118-49b5-b87d-85206559248f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:21.533502 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-5b233ef2-c118-49b5-b87d-85206559248f None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:34:21.534715 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-5b233ef2-c118-49b5-b87d-85206559248f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:21.593176 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1181/2374] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:34:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:34:21.601523 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f783ffa-fe62-444b-8498-9c6933b51342 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:21.601808 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f783ffa-fe62-444b-8498-9c6933b51342 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:21.602103 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f783ffa-fe62-444b-8498-9c6933b51342 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:21.622403 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-9f783ffa-fe62-444b-8498-9c6933b51342 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:34:21.623592 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-9f783ffa-fe62-444b-8498-9c6933b51342 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:21.631679 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1194/2375] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:34:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:34:21.635867 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b233ef2-c118-49b5-b87d-85206559248f None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:21.645877 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b233ef2-c118-49b5-b87d-85206559248f None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:21.650975 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a33459f-cecd-4c4f-95b3-7c9a4f6c0b90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:21.651573 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a33459f-cecd-4c4f-95b3-7c9a4f6c0b90 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:21.651848 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a33459f-cecd-4c4f-95b3-7c9a4f6c0b90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:21.652198 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a33459f-cecd-4c4f-95b3-7c9a4f6c0b90 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:21.652835 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a33459f-cecd-4c4f-95b3-7c9a4f6c0b90 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:34:21.653164 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a33459f-cecd-4c4f-95b3-7c9a4f6c0b90 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:21.654748 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a33459f-cecd-4c4f-95b3-7c9a4f6c0b90 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:21.669637 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1182/2376] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:34:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:34:21.671779 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f783ffa-fe62-444b-8498-9c6933b51342 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:21.682075 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f783ffa-fe62-444b-8498-9c6933b51342 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:21.686725 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-761d110d-7492-48e9-a642-1f3d1b0f60d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:21.687317 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761d110d-7492-48e9-a642-1f3d1b0f60d2 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:21.687596 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761d110d-7492-48e9-a642-1f3d1b0f60d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:21.687863 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761d110d-7492-48e9-a642-1f3d1b0f60d2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:21.688490 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-761d110d-7492-48e9-a642-1f3d1b0f60d2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:21.690450 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-761d110d-7492-48e9-a642-1f3d1b0f60d2 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:21.718257 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1195/2377] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:34:21 2022] POST /identity/v3/projects => generated 375 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:34:21.722465 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a33459f-cecd-4c4f-95b3-7c9a4f6c0b90 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:21.732893 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a33459f-cecd-4c4f-95b3-7c9a4f6c0b90 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:21.753639 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:DELETE=9,keystone:SELECT=90,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:34:21.792120 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0133d0bb-de52-488b-bf17-f1aa0295a560 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:21.792733 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0133d0bb-de52-488b-bf17-f1aa0295a560 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:21.793011 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0133d0bb-de52-488b-bf17-f1aa0295a560 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:21.793315 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0133d0bb-de52-488b-bf17-f1aa0295a560 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:21.793997 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0133d0bb-de52-488b-bf17-f1aa0295a560 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:21.796263 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0133d0bb-de52-488b-bf17-f1aa0295a560 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:21.827249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1183/2378] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:34:21 2022] POST /identity/v3/users => generated 288 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:34:21.828424 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-761d110d-7492-48e9-a642-1f3d1b0f60d2 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:21.839349 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-761d110d-7492-48e9-a642-1f3d1b0f60d2 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:21.844068 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ff9d91-98ff-4721-8617-54322a44ba4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:21.844673 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ff9d91-98ff-4721-8617-54322a44ba4a admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:21.844955 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ff9d91-98ff-4721-8617-54322a44ba4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:21.845276 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ff9d91-98ff-4721-8617-54322a44ba4a admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:21.846038 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ff9d91-98ff-4721-8617-54322a44ba4a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:21.847697 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ff9d91-98ff-4721-8617-54322a44ba4a admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:21.858966 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1196/2379] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:34:21 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:34:21.862876 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0133d0bb-de52-488b-bf17-f1aa0295a560 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:21.873660 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0133d0bb-de52-488b-bf17-f1aa0295a560 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:21.879491 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216d4575-f320-4566-b342-0c2060ed03c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:21.880106 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216d4575-f320-4566-b342-0c2060ed03c9 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:21.880574 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216d4575-f320-4566-b342-0c2060ed03c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:21.880847 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216d4575-f320-4566-b342-0c2060ed03c9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:21.881533 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-216d4575-f320-4566-b342-0c2060ed03c9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:21.883264 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-216d4575-f320-4566-b342-0c2060ed03c9 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:21.894328 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1184/2380] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:34:21 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 27 01:34:21.898991 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ff9d91-98ff-4721-8617-54322a44ba4a admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:21.909241 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ff9d91-98ff-4721-8617-54322a44ba4a admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:21.914468 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21307d95-c1da-4071-a951-b6522dc41f19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:21.915114 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21307d95-c1da-4071-a951-b6522dc41f19 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:21.915387 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21307d95-c1da-4071-a951-b6522dc41f19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:21.915655 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21307d95-c1da-4071-a951-b6522dc41f19 admin admin] PATH_INFO: `/v3/projects/8745ff179186480dba10bd18077ebf02/users/d99b0cd7ef1c4867a4dcb012c64a90c4/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:21.932369 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21307d95-c1da-4071-a951-b6522dc41f19 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8745ff179186480dba10bd18077ebf02, user_id=d99b0cd7ef1c4867a4dcb012c64a90c4, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:21.935086 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21307d95-c1da-4071-a951-b6522dc41f19 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:21.951047 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1197/2381] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:34:21 2022] PUT /identity/v3/projects/8745ff179186480dba10bd18077ebf02/users/d99b0cd7ef1c4867a4dcb012c64a90c4/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:34:21.961779 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0057e48-89a7-4e51-a6d4-a980434736d2 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:21.962065 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0057e48-89a7-4e51-a6d4-a980434736d2 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:21.962343 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0057e48-89a7-4e51-a6d4-a980434736d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:21.991898 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-e0057e48-89a7-4e51-a6d4-a980434736d2 None None] MFA Rules not processed for user `d99b0cd7ef1c4867a4dcb012c64a90c4`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:34:21.993138 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e0057e48-89a7-4e51-a6d4-a980434736d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:22.059907 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1185/2382] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:34:21 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:34:22.071291 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21307d95-c1da-4071-a951-b6522dc41f19 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:22.073102 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-21307d95-c1da-4071-a951-b6522dc41f19 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:22.132176 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21307d95-c1da-4071-a951-b6522dc41f19 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:22.136202 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62105d8-26f7-46c8-a03e-ccc25b2a8706 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:22.136817 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62105d8-26f7-46c8-a03e-ccc25b2a8706 service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:22.137112 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62105d8-26f7-46c8-a03e-ccc25b2a8706 service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:22.137381 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62105d8-26f7-46c8-a03e-ccc25b2a8706 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:22.146902 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62105d8-26f7-46c8-a03e-ccc25b2a8706 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:22.148356 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62105d8-26f7-46c8-a03e-ccc25b2a8706 service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:22.157726 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1198/2383] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:34:22 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:34:25.198854 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0057e48-89a7-4e51-a6d4-a980434736d2 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:25.261534 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0057e48-89a7-4e51-a6d4-a980434736d2 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:25.266021 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e48ee9-c59c-4fba-b301-ba6b4b3030d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:25.266651 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e48ee9-c59c-4fba-b301-ba6b4b3030d1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:25.266924 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e48ee9-c59c-4fba-b301-ba6b4b3030d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:25.267193 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e48ee9-c59c-4fba-b301-ba6b4b3030d1 admin admin] PATH_INFO: `/v3/users/d99b0cd7ef1c4867a4dcb012c64a90c4` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:25.268365 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e48ee9-c59c-4fba-b301-ba6b4b3030d1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d99b0cd7ef1c4867a4dcb012c64a90c4)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:25.270341 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e48ee9-c59c-4fba-b301-ba6b4b3030d1 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:25.310093 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=92,keystone:INSERT=3,keystone:DELETE=5 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:34:25.322563 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-62e48ee9-c59c-4fba-b301-ba6b4b3030d1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd99b0cd7ef1c4867a4dcb012c64a90c4'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:25.329825 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-62e48ee9-c59c-4fba-b301-ba6b4b3030d1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd99b0cd7ef1c4867a4dcb012c64a90c4'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:25.330546 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-62e48ee9-c59c-4fba-b301-ba6b4b3030d1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd99b0cd7ef1c4867a4dcb012c64a90c4'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:25.342690 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-62e48ee9-c59c-4fba-b301-ba6b4b3030d1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd99b0cd7ef1c4867a4dcb012c64a90c4'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:25.349895 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-62e48ee9-c59c-4fba-b301-ba6b4b3030d1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd99b0cd7ef1c4867a4dcb012c64a90c4'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:25.355628 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1186/2384] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:34:25 2022] DELETE /identity/v3/users/d99b0cd7ef1c4867a4dcb012c64a90c4 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:34:25.360515 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70202634-3c71-4544-84c9-3b37475f46b3 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:25.361018 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70202634-3c71-4544-84c9-3b37475f46b3 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:25.361146 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70202634-3c71-4544-84c9-3b37475f46b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:25.380486 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-70202634-3c71-4544-84c9-3b37475f46b3 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:34:25.381744 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-70202634-3c71-4544-84c9-3b37475f46b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:25.442447 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1199/2385] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:34:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:34:25.452218 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e48ee9-c59c-4fba-b301-ba6b4b3030d1 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:25.453673 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-62e48ee9-c59c-4fba-b301-ba6b4b3030d1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:25.512873 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e48ee9-c59c-4fba-b301-ba6b4b3030d1 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:25.516538 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a5290e-886b-4e15-a712-c17fbfc12849 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:25.517174 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a5290e-886b-4e15-a712-c17fbfc12849 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:25.517476 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a5290e-886b-4e15-a712-c17fbfc12849 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:25.517748 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a5290e-886b-4e15-a712-c17fbfc12849 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:25.528475 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a5290e-886b-4e15-a712-c17fbfc12849 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:25.530001 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a5290e-886b-4e15-a712-c17fbfc12849 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:25.541040 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1187/2386] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:34:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:34:25.826967 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70202634-3c71-4544-84c9-3b37475f46b3 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:25.828383 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-70202634-3c71-4544-84c9-3b37475f46b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:25.841042 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70202634-3c71-4544-84c9-3b37475f46b3 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:25.848205 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674ce641-88b8-4310-aa89-2ea58d14e54f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:25.848865 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674ce641-88b8-4310-aa89-2ea58d14e54f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:25.849757 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674ce641-88b8-4310-aa89-2ea58d14e54f admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:25.850030 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674ce641-88b8-4310-aa89-2ea58d14e54f admin admin] PATH_INFO: `/v3/projects/8745ff179186480dba10bd18077ebf02` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:25.851237 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-674ce641-88b8-4310-aa89-2ea58d14e54f admin admin] RBAC: Authorizing `identity:delete_project(project_id=8745ff179186480dba10bd18077ebf02)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:25.853223 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-674ce641-88b8-4310-aa89-2ea58d14e54f admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:25.866111 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:INSERT=3,keystone:DELETE=3,keystone:SELECT=94 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:34:25.905049 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-674ce641-88b8-4310-aa89-2ea58d14e54f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8745ff179186480dba10bd18077ebf02', 'request_id': 'req-674ce641-88b8-4310-aa89-2ea58d14e54f'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:25.913453 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-674ce641-88b8-4310-aa89-2ea58d14e54f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8745ff179186480dba10bd18077ebf02', 'request_id': 'req-674ce641-88b8-4310-aa89-2ea58d14e54f'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:25.919848 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-674ce641-88b8-4310-aa89-2ea58d14e54f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8745ff179186480dba10bd18077ebf02', 'request_id': 'req-674ce641-88b8-4310-aa89-2ea58d14e54f'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:25.930675 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-674ce641-88b8-4310-aa89-2ea58d14e54f admin admin] The token cache is being invalidate because project 8745ff179186480dba10bd18077ebf02 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:34:25.930968 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-674ce641-88b8-4310-aa89-2ea58d14e54f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:25.931643 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-674ce641-88b8-4310-aa89-2ea58d14e54f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:25.933183 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1200/2387] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:34:25 2022] DELETE /identity/v3/projects/8745ff179186480dba10bd18077ebf02 => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:34:25.938068 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89773540-f4c8-4b66-a9f3-39c5092762d7 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:25.938350 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89773540-f4c8-4b66-a9f3-39c5092762d7 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:25.938653 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89773540-f4c8-4b66-a9f3-39c5092762d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:25.960957 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-89773540-f4c8-4b66-a9f3-39c5092762d7 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:34:25.962141 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-89773540-f4c8-4b66-a9f3-39c5092762d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:26.024551 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1188/2388] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:34:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:34:26.091081 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4186917-d83b-4e11-b7cb-ac40f3dfcfd4 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:26.091360 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4186917-d83b-4e11-b7cb-ac40f3dfcfd4 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:26.091625 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4186917-d83b-4e11-b7cb-ac40f3dfcfd4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:26.111710 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-c4186917-d83b-4e11-b7cb-ac40f3dfcfd4 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:34:26.112897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-c4186917-d83b-4e11-b7cb-ac40f3dfcfd4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:26.120900 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1201/2389] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:34:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:34:26.125392 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89773540-f4c8-4b66-a9f3-39c5092762d7 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:26.136055 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89773540-f4c8-4b66-a9f3-39c5092762d7 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:26.140670 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44d98cc-a502-413a-90b1-47c32d6a553a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:26.141327 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44d98cc-a502-413a-90b1-47c32d6a553a admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:26.141730 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44d98cc-a502-413a-90b1-47c32d6a553a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:26.142002 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44d98cc-a502-413a-90b1-47c32d6a553a admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:26.142650 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44d98cc-a502-413a-90b1-47c32d6a553a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:34:26.142946 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44d98cc-a502-413a-90b1-47c32d6a553a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:26.144536 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44d98cc-a502-413a-90b1-47c32d6a553a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:26.160274 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1189/2390] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:34:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:34:26.162099 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4186917-d83b-4e11-b7cb-ac40f3dfcfd4 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:26.171080 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4186917-d83b-4e11-b7cb-ac40f3dfcfd4 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:26.175910 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2eeb020-47b0-4927-82c9-4e2c570012a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:26.176496 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2eeb020-47b0-4927-82c9-4e2c570012a0 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:26.176772 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2eeb020-47b0-4927-82c9-4e2c570012a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:26.177038 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2eeb020-47b0-4927-82c9-4e2c570012a0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:26.177679 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2eeb020-47b0-4927-82c9-4e2c570012a0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:26.179630 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2eeb020-47b0-4927-82c9-4e2c570012a0 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:26.207319 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1202/2391] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:34:26 2022] POST /identity/v3/projects => generated 371 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:34:26.212204 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44d98cc-a502-413a-90b1-47c32d6a553a admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:26.221825 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44d98cc-a502-413a-90b1-47c32d6a553a admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:26.264694 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:34:26.283262 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05532a9c-7a98-452f-8589-c80142cf7781 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:26.284180 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05532a9c-7a98-452f-8589-c80142cf7781 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:26.284556 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05532a9c-7a98-452f-8589-c80142cf7781 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:26.285005 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05532a9c-7a98-452f-8589-c80142cf7781 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:26.285745 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05532a9c-7a98-452f-8589-c80142cf7781 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:26.287671 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05532a9c-7a98-452f-8589-c80142cf7781 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:26.310461 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1190/2392] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:34:26 2022] POST /identity/v3/users => generated 286 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:34:26.314531 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2eeb020-47b0-4927-82c9-4e2c570012a0 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:26.326439 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2eeb020-47b0-4927-82c9-4e2c570012a0 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:26.330960 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58850541-dbee-4e1d-8e83-0755efff58da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:26.331542 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58850541-dbee-4e1d-8e83-0755efff58da admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:26.331815 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58850541-dbee-4e1d-8e83-0755efff58da admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:26.332081 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58850541-dbee-4e1d-8e83-0755efff58da admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:26.332707 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58850541-dbee-4e1d-8e83-0755efff58da admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:26.334345 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58850541-dbee-4e1d-8e83-0755efff58da admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:26.345283 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1203/2393] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:34:26 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:34:26.349320 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05532a9c-7a98-452f-8589-c80142cf7781 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:26.360236 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05532a9c-7a98-452f-8589-c80142cf7781 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:26.365086 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0211b21a-61ec-4441-8e09-89bc3c598328 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:26.365666 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0211b21a-61ec-4441-8e09-89bc3c598328 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:26.365945 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0211b21a-61ec-4441-8e09-89bc3c598328 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:26.366210 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0211b21a-61ec-4441-8e09-89bc3c598328 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:26.366839 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0211b21a-61ec-4441-8e09-89bc3c598328 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:26.369538 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0211b21a-61ec-4441-8e09-89bc3c598328 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:26.380412 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1191/2394] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:34:26 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:34:26.385671 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58850541-dbee-4e1d-8e83-0755efff58da admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:26.397535 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58850541-dbee-4e1d-8e83-0755efff58da admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:26.402991 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83ed34b-7cb5-4c4a-b95d-3eb8a695ee93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:26.403641 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83ed34b-7cb5-4c4a-b95d-3eb8a695ee93 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:26.403913 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83ed34b-7cb5-4c4a-b95d-3eb8a695ee93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:26.404181 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83ed34b-7cb5-4c4a-b95d-3eb8a695ee93 admin admin] PATH_INFO: `/v3/projects/4d885b933eca4da0a9ed5474f84ccfac/users/180a1773229e406dab4e44160f793610/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:26.419814 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c83ed34b-7cb5-4c4a-b95d-3eb8a695ee93 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4d885b933eca4da0a9ed5474f84ccfac, user_id=180a1773229e406dab4e44160f793610, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:26.422781 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c83ed34b-7cb5-4c4a-b95d-3eb8a695ee93 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:26.436955 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1204/2395] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:34:26 2022] PUT /identity/v3/projects/4d885b933eca4da0a9ed5474f84ccfac/users/180a1773229e406dab4e44160f793610/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:34:26.447861 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b043b8-a7ac-4a43-852a-40e8ac3ecd6e None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:26.448138 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b043b8-a7ac-4a43-852a-40e8ac3ecd6e None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:26.448407 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b043b8-a7ac-4a43-852a-40e8ac3ecd6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:26.476553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-a4b043b8-a7ac-4a43-852a-40e8ac3ecd6e None None] MFA Rules not processed for user `180a1773229e406dab4e44160f793610`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:34:26.478608 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-a4b043b8-a7ac-4a43-852a-40e8ac3ecd6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:26.547446 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1192/2396] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:34:26 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:34:26.558719 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83ed34b-7cb5-4c4a-b95d-3eb8a695ee93 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:26.560446 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-c83ed34b-7cb5-4c4a-b95d-3eb8a695ee93 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:26.622152 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83ed34b-7cb5-4c4a-b95d-3eb8a695ee93 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:26.625995 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ad2825-f1b2-42e9-8fef-5a53040828f4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:26.626608 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ad2825-f1b2-42e9-8fef-5a53040828f4 service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:26.626885 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ad2825-f1b2-42e9-8fef-5a53040828f4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:26.627151 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ad2825-f1b2-42e9-8fef-5a53040828f4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:26.636595 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ad2825-f1b2-42e9-8fef-5a53040828f4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:26.638120 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ad2825-f1b2-42e9-8fef-5a53040828f4 service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:26.646906 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1205/2397] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:34:26 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:34:26.672615 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b043b8-a7ac-4a43-852a-40e8ac3ecd6e None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:26.738081 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b043b8-a7ac-4a43-852a-40e8ac3ecd6e None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:26.743148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3154c49e-24a3-47cf-a37d-a879d8d01fbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:26.744393 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3154c49e-24a3-47cf-a37d-a879d8d01fbc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:26.744393 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3154c49e-24a3-47cf-a37d-a879d8d01fbc admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:26.744393 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3154c49e-24a3-47cf-a37d-a879d8d01fbc admin admin] PATH_INFO: `/v3/users/180a1773229e406dab4e44160f793610` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:26.745867 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3154c49e-24a3-47cf-a37d-a879d8d01fbc admin admin] RBAC: Authorizing `identity:delete_user(user_id=180a1773229e406dab4e44160f793610)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:26.747898 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3154c49e-24a3-47cf-a37d-a879d8d01fbc admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:26.798013 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3154c49e-24a3-47cf-a37d-a879d8d01fbc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '180a1773229e406dab4e44160f793610'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:26.809073 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=88,keystone:INSERT=4,keystone:DELETE=8 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:34:26.819596 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3154c49e-24a3-47cf-a37d-a879d8d01fbc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '180a1773229e406dab4e44160f793610'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:26.819865 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3154c49e-24a3-47cf-a37d-a879d8d01fbc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '180a1773229e406dab4e44160f793610'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:26.835688 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3154c49e-24a3-47cf-a37d-a879d8d01fbc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '180a1773229e406dab4e44160f793610'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:26.843474 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3154c49e-24a3-47cf-a37d-a879d8d01fbc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '180a1773229e406dab4e44160f793610'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:26.849654 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1193/2398] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:34:26 2022] DELETE /identity/v3/users/180a1773229e406dab4e44160f793610 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:34:26.854208 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3362442c-bf52-426b-80b9-b46a919460ee None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:26.854494 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3362442c-bf52-426b-80b9-b46a919460ee None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:26.854770 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3362442c-bf52-426b-80b9-b46a919460ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:26.875238 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-3362442c-bf52-426b-80b9-b46a919460ee None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:34:26.876494 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-3362442c-bf52-426b-80b9-b46a919460ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:26.910493 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:34:26.940647 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1206/2399] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:34:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:34:26.951416 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3154c49e-24a3-47cf-a37d-a879d8d01fbc admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:26.952837 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-3154c49e-24a3-47cf-a37d-a879d8d01fbc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:27.011346 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3154c49e-24a3-47cf-a37d-a879d8d01fbc admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:27.014898 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba26b19-4661-4efc-a8b4-c0c36435a4cd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:27.015508 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba26b19-4661-4efc-a8b4-c0c36435a4cd service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:27.015786 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba26b19-4661-4efc-a8b4-c0c36435a4cd service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:27.016054 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba26b19-4661-4efc-a8b4-c0c36435a4cd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:27.025509 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba26b19-4661-4efc-a8b4-c0c36435a4cd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:27.026942 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba26b19-4661-4efc-a8b4-c0c36435a4cd service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:27.036488 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1194/2400] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:34:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:34:27.317266 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3362442c-bf52-426b-80b9-b46a919460ee None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:27.318933 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-3362442c-bf52-426b-80b9-b46a919460ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:27.331548 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3362442c-bf52-426b-80b9-b46a919460ee None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:27.336052 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5461d3ff-248c-47bb-b2ba-e40a6fe3eb36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:27.337402 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5461d3ff-248c-47bb-b2ba-e40a6fe3eb36 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:27.337682 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5461d3ff-248c-47bb-b2ba-e40a6fe3eb36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:27.338148 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5461d3ff-248c-47bb-b2ba-e40a6fe3eb36 admin admin] PATH_INFO: `/v3/projects/4d885b933eca4da0a9ed5474f84ccfac` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:27.339295 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5461d3ff-248c-47bb-b2ba-e40a6fe3eb36 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4d885b933eca4da0a9ed5474f84ccfac)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:27.341254 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5461d3ff-248c-47bb-b2ba-e40a6fe3eb36 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:27.390915 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-5461d3ff-248c-47bb-b2ba-e40a6fe3eb36 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4d885b933eca4da0a9ed5474f84ccfac', 'request_id': 'req-5461d3ff-248c-47bb-b2ba-e40a6fe3eb36'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:27.398436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-5461d3ff-248c-47bb-b2ba-e40a6fe3eb36 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4d885b933eca4da0a9ed5474f84ccfac', 'request_id': 'req-5461d3ff-248c-47bb-b2ba-e40a6fe3eb36'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:27.405347 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-5461d3ff-248c-47bb-b2ba-e40a6fe3eb36 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4d885b933eca4da0a9ed5474f84ccfac', 'request_id': 'req-5461d3ff-248c-47bb-b2ba-e40a6fe3eb36'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:27.420403 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-5461d3ff-248c-47bb-b2ba-e40a6fe3eb36 admin admin] The token cache is being invalidate because project 4d885b933eca4da0a9ed5474f84ccfac was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:34:27.420691 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-5461d3ff-248c-47bb-b2ba-e40a6fe3eb36 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:27.421375 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-5461d3ff-248c-47bb-b2ba-e40a6fe3eb36 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:34:27.423142 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1207/2401] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:34:27 2022] DELETE /identity/v3/projects/4d885b933eca4da0a9ed5474f84ccfac => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:34:27.428265 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa42a16b-6277-4ace-b87b-b6dbf87ee13d None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:27.428730 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa42a16b-6277-4ace-b87b-b6dbf87ee13d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:27.428842 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa42a16b-6277-4ace-b87b-b6dbf87ee13d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:27.448754 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-fa42a16b-6277-4ace-b87b-b6dbf87ee13d None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:34:27.450033 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-fa42a16b-6277-4ace-b87b-b6dbf87ee13d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:27.513770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1195/2402] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:34:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:34:27.521633 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a67424-fcd2-4d8e-922b-cd3a38926cea None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:27.521915 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a67424-fcd2-4d8e-922b-cd3a38926cea None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:27.522185 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a67424-fcd2-4d8e-922b-cd3a38926cea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:27.547999 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-c0a67424-fcd2-4d8e-922b-cd3a38926cea None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:34:27.549262 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-c0a67424-fcd2-4d8e-922b-cd3a38926cea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:27.557818 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1208/2403] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:34:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:34:27.562174 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa42a16b-6277-4ace-b87b-b6dbf87ee13d None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:27.572299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa42a16b-6277-4ace-b87b-b6dbf87ee13d None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:27.578616 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9fd9ee6-b318-496a-93aa-983aa7a87258 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:27.579214 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fd9ee6-b318-496a-93aa-983aa7a87258 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:27.579495 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fd9ee6-b318-496a-93aa-983aa7a87258 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:27.579764 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fd9ee6-b318-496a-93aa-983aa7a87258 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:27.580441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9fd9ee6-b318-496a-93aa-983aa7a87258 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:34:27.580743 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9fd9ee6-b318-496a-93aa-983aa7a87258 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:27.582374 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9fd9ee6-b318-496a-93aa-983aa7a87258 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:27.596740 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1196/2404] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:34:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 27 01:34:27.603748 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a67424-fcd2-4d8e-922b-cd3a38926cea None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:27.612937 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a67424-fcd2-4d8e-922b-cd3a38926cea None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:27.618854 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a4506f-e02a-4134-adeb-4c76487ad8d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:27.619447 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a4506f-e02a-4134-adeb-4c76487ad8d0 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:27.619727 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a4506f-e02a-4134-adeb-4c76487ad8d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:27.619992 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a4506f-e02a-4134-adeb-4c76487ad8d0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:27.620616 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6a4506f-e02a-4134-adeb-4c76487ad8d0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:27.622563 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6a4506f-e02a-4134-adeb-4c76487ad8d0 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:27.649870 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1209/2405] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:34:27 2022] POST /identity/v3/projects => generated 385 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:34:27.654287 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9fd9ee6-b318-496a-93aa-983aa7a87258 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:27.665432 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9fd9ee6-b318-496a-93aa-983aa7a87258 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:27.735284 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4606db-093b-4f0c-bb10-8746c4ce98cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:27.735895 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4606db-093b-4f0c-bb10-8746c4ce98cb admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:27.736172 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4606db-093b-4f0c-bb10-8746c4ce98cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:27.736441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4606db-093b-4f0c-bb10-8746c4ce98cb admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:27.737111 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4606db-093b-4f0c-bb10-8746c4ce98cb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:27.739857 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4606db-093b-4f0c-bb10-8746c4ce98cb admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:27.748303 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:34:27.762840 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1197/2406] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:34:27 2022] POST /identity/v3/users => generated 293 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:34:27.768040 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a4506f-e02a-4134-adeb-4c76487ad8d0 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:27.779672 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a4506f-e02a-4134-adeb-4c76487ad8d0 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:27.784484 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003efbc7-9f52-4879-910c-cf9ce2e6d12d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:27.785109 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003efbc7-9f52-4879-910c-cf9ce2e6d12d admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:27.785396 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003efbc7-9f52-4879-910c-cf9ce2e6d12d admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:27.785666 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003efbc7-9f52-4879-910c-cf9ce2e6d12d admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:27.786301 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-003efbc7-9f52-4879-910c-cf9ce2e6d12d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:27.787905 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-003efbc7-9f52-4879-910c-cf9ce2e6d12d admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:27.800931 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1210/2407] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:34:27 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:34:27.805746 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4606db-093b-4f0c-bb10-8746c4ce98cb admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:27.821256 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4606db-093b-4f0c-bb10-8746c4ce98cb admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:27.828301 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a47a33f-1531-49e6-88cf-896fc8dbea33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:27.828935 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a47a33f-1531-49e6-88cf-896fc8dbea33 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:27.829254 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a47a33f-1531-49e6-88cf-896fc8dbea33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:27.829523 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a47a33f-1531-49e6-88cf-896fc8dbea33 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:27.830178 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a47a33f-1531-49e6-88cf-896fc8dbea33 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:27.832039 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a47a33f-1531-49e6-88cf-896fc8dbea33 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:27.847885 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1198/2408] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:34:27 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:34:27.853623 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003efbc7-9f52-4879-910c-cf9ce2e6d12d admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:27.865548 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003efbc7-9f52-4879-910c-cf9ce2e6d12d admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:27.870574 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d52ab0bf-62d3-440e-9909-19f304629c0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:27.872103 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52ab0bf-62d3-440e-9909-19f304629c0d admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:27.872103 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52ab0bf-62d3-440e-9909-19f304629c0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:27.872103 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52ab0bf-62d3-440e-9909-19f304629c0d admin admin] PATH_INFO: `/v3/projects/9e496297038149beb8bfe69c53234988/users/e1ff3076fbb342caaf7d617c079a9080/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:27.888806 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d52ab0bf-62d3-440e-9909-19f304629c0d admin admin] RBAC: Authorizing `identity:create_grant(project_id=9e496297038149beb8bfe69c53234988, user_id=e1ff3076fbb342caaf7d617c079a9080, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:27.891528 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d52ab0bf-62d3-440e-9909-19f304629c0d admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:27.906383 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1211/2409] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:34:27 2022] PUT /identity/v3/projects/9e496297038149beb8bfe69c53234988/users/e1ff3076fbb342caaf7d617c079a9080/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:34:27.913124 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34455dfa-742f-4168-acda-a5d04cc06dc1 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:27.913410 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34455dfa-742f-4168-acda-a5d04cc06dc1 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:27.913680 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34455dfa-742f-4168-acda-a5d04cc06dc1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:27.938226 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-34455dfa-742f-4168-acda-a5d04cc06dc1 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:34:27.939441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-34455dfa-742f-4168-acda-a5d04cc06dc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:28.009704 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1199/2410] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:34:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:34:28.019733 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d52ab0bf-62d3-440e-9909-19f304629c0d admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:28.021504 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-d52ab0bf-62d3-440e-9909-19f304629c0d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:28.081761 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d52ab0bf-62d3-440e-9909-19f304629c0d admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:28.085380 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7664889b-208b-45f2-b0e6-0409d4dc65aa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:28.086006 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7664889b-208b-45f2-b0e6-0409d4dc65aa service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:28.086359 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7664889b-208b-45f2-b0e6-0409d4dc65aa service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:28.086632 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7664889b-208b-45f2-b0e6-0409d4dc65aa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:28.097904 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7664889b-208b-45f2-b0e6-0409d4dc65aa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:28.100097 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7664889b-208b-45f2-b0e6-0409d4dc65aa service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:28.122344 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1212/2411] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:34:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:34:32.291423 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308b2ece-824f-4672-8d27-b3b0c86fb647 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:32.291981 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308b2ece-824f-4672-8d27-b3b0c86fb647 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:32.291981 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308b2ece-824f-4672-8d27-b3b0c86fb647 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:32.324237 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-308b2ece-824f-4672-8d27-b3b0c86fb647 None None] MFA Rules not processed for user `e1ff3076fbb342caaf7d617c079a9080`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:34:32.325662 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-308b2ece-824f-4672-8d27-b3b0c86fb647 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:32.396214 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1200/2412] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:34:32 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:34:32.401328 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7664889b-208b-45f2-b0e6-0409d4dc65aa service neutron] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:32.406509 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:34:32.413711 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7664889b-208b-45f2-b0e6-0409d4dc65aa service neutron] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:32.420162 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419f6764-461c-46aa-aea6-c21661eec713 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:32.420776 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419f6764-461c-46aa-aea6-c21661eec713 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:32.421130 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419f6764-461c-46aa-aea6-c21661eec713 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:32.421469 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419f6764-461c-46aa-aea6-c21661eec713 admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:32.422424 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-419f6764-461c-46aa-aea6-c21661eec713 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:32.424712 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-419f6764-461c-46aa-aea6-c21661eec713 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:32.473045 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1213/2413] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:34:32 2022] POST /identity/v3/projects => generated 385 bytes in 72 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:34:32.477320 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308b2ece-824f-4672-8d27-b3b0c86fb647 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:32.493873 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308b2ece-824f-4672-8d27-b3b0c86fb647 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:32.556638 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ad4748-5f06-42db-aed9-7a8917e63786 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:32.557324 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ad4748-5f06-42db-aed9-7a8917e63786 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:32.557604 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ad4748-5f06-42db-aed9-7a8917e63786 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:32.557880 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ad4748-5f06-42db-aed9-7a8917e63786 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:32.558525 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ad4748-5f06-42db-aed9-7a8917e63786 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:32.561680 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ad4748-5f06-42db-aed9-7a8917e63786 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:32.579303 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:INSERT=6,keystone:SELECT=94 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:34:32.586683 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1201/2414] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:34:32 2022] POST /identity/v3/users => generated 299 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:34:32.590674 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419f6764-461c-46aa-aea6-c21661eec713 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:32.601358 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419f6764-461c-46aa-aea6-c21661eec713 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:32.606319 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81abbb80-76c1-4bcf-ab43-b9069368ff37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:32.606932 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81abbb80-76c1-4bcf-ab43-b9069368ff37 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:32.607214 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81abbb80-76c1-4bcf-ab43-b9069368ff37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:32.607484 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81abbb80-76c1-4bcf-ab43-b9069368ff37 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:32.608127 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81abbb80-76c1-4bcf-ab43-b9069368ff37 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:32.609794 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81abbb80-76c1-4bcf-ab43-b9069368ff37 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:32.621732 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1214/2415] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:34:32 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:34:32.626028 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ad4748-5f06-42db-aed9-7a8917e63786 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:32.637358 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ad4748-5f06-42db-aed9-7a8917e63786 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:32.642136 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71352759-ee61-4c2e-b8b6-203e2467bddd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:32.642793 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71352759-ee61-4c2e-b8b6-203e2467bddd admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:32.643069 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71352759-ee61-4c2e-b8b6-203e2467bddd admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:32.643340 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71352759-ee61-4c2e-b8b6-203e2467bddd admin admin] PATH_INFO: `/v3/projects/3d3b0ef54598449e9b3430810d6a081e/users/12e68c094a8f4814a06d3a6143746982/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:32.659709 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71352759-ee61-4c2e-b8b6-203e2467bddd admin admin] RBAC: Authorizing `identity:create_grant(project_id=3d3b0ef54598449e9b3430810d6a081e, user_id=12e68c094a8f4814a06d3a6143746982, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:32.662426 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71352759-ee61-4c2e-b8b6-203e2467bddd admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:32.677551 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1202/2416] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:34:32 2022] PUT /identity/v3/projects/3d3b0ef54598449e9b3430810d6a081e/users/12e68c094a8f4814a06d3a6143746982/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:34:32.682159 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81abbb80-76c1-4bcf-ab43-b9069368ff37 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:32.692419 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81abbb80-76c1-4bcf-ab43-b9069368ff37 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:32.752402 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff508da3-89d6-4b17-be0d-9bbcbf028cfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:32.753015 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff508da3-89d6-4b17-be0d-9bbcbf028cfb admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:32.753392 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff508da3-89d6-4b17-be0d-9bbcbf028cfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:32.753661 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff508da3-89d6-4b17-be0d-9bbcbf028cfb admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:32.754299 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff508da3-89d6-4b17-be0d-9bbcbf028cfb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:32.755898 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff508da3-89d6-4b17-be0d-9bbcbf028cfb admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:32.769566 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1215/2417] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:34:32 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:34:32.770776 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71352759-ee61-4c2e-b8b6-203e2467bddd admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:32.782256 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71352759-ee61-4c2e-b8b6-203e2467bddd admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:32.787573 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca8ad94-8590-4c51-9529-b9be0c3b8818 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:32.788272 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca8ad94-8590-4c51-9529-b9be0c3b8818 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:32.788552 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca8ad94-8590-4c51-9529-b9be0c3b8818 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:32.788828 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca8ad94-8590-4c51-9529-b9be0c3b8818 admin admin] PATH_INFO: `/v3/projects/3d3b0ef54598449e9b3430810d6a081e/users/12e68c094a8f4814a06d3a6143746982/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:32.791345 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca8ad94-8590-4c51-9529-b9be0c3b8818 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3d3b0ef54598449e9b3430810d6a081e, user_id=12e68c094a8f4814a06d3a6143746982, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:32.794063 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca8ad94-8590-4c51-9529-b9be0c3b8818 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:32.807262 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1203/2418] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:34:32 2022] PUT /identity/v3/projects/3d3b0ef54598449e9b3430810d6a081e/users/12e68c094a8f4814a06d3a6143746982/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:34:37.183812 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88d82cf-d963-432a-8ec2-5f0774525d20 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:37.184385 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88d82cf-d963-432a-8ec2-5f0774525d20 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:37.184385 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88d82cf-d963-432a-8ec2-5f0774525d20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:37.214105 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-f88d82cf-d963-432a-8ec2-5f0774525d20 None None] MFA Rules not processed for user `12e68c094a8f4814a06d3a6143746982`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:34:37.215290 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-f88d82cf-d963-432a-8ec2-5f0774525d20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:37.285766 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1216/2419] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:34:37 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:34:37.299339 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca8ad94-8590-4c51-9529-b9be0c3b8818 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:37.301034 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-3ca8ad94-8590-4c51-9529-b9be0c3b8818 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:37.358851 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca8ad94-8590-4c51-9529-b9be0c3b8818 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:37.362416 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7594ba-5f28-4382-8b5a-cb80ba4ad290 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2880dbb00eec471285d242e8183dd8d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:37.363022 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7594ba-5f28-4382-8b5a-cb80ba4ad290 service glance] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:37.363298 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7594ba-5f28-4382-8b5a-cb80ba4ad290 service glance] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:37.363571 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7594ba-5f28-4382-8b5a-cb80ba4ad290 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:37.422914 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec7594ba-5f28-4382-8b5a-cb80ba4ad290 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:37.424350 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec7594ba-5f28-4382-8b5a-cb80ba4ad290 service glance] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:37.434576 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1204/2420] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:34:37 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:34:37.592743 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae79c5c-b945-4d34-a791-a712041a1031 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:37.593028 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae79c5c-b945-4d34-a791-a712041a1031 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:37.593509 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae79c5c-b945-4d34-a791-a712041a1031 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:37.648668 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-9ae79c5c-b945-4d34-a791-a712041a1031 None None] MFA Rules not processed for user `7a3f8b5eb09f4ba29f4bd343bd91f6a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:34:37.649965 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-9ae79c5c-b945-4d34-a791-a712041a1031 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:37.679526 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:34:37.747480 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1217/2421] 10.222.0.42 () {58 vars in 1009 bytes} [Thu Oct 27 01:34:37 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:34:37.769860 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7594ba-5f28-4382-8b5a-cb80ba4ad290 service glance] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:37.771286 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ec7594ba-5f28-4382-8b5a-cb80ba4ad290 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:37.870560 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:34:37.881300 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7594ba-5f28-4382-8b5a-cb80ba4ad290 service glance] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:37.887634 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17b7f8f6-1716-434c-bb8a-705c8b56402c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77f2cf9e0c3541f9bccfb7e213d88609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:37.888424 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b7f8f6-1716-434c-bb8a-705c8b56402c service swift] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:37.888711 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b7f8f6-1716-434c-bb8a-705c8b56402c service swift] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:37.888984 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b7f8f6-1716-434c-bb8a-705c8b56402c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:37.897988 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17b7f8f6-1716-434c-bb8a-705c8b56402c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:37.899899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17b7f8f6-1716-434c-bb8a-705c8b56402c service swift] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:37.908757 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1205/2422] 10.222.0.42 () {60 vars in 1384 bytes} [Thu Oct 27 01:34:37 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 27 01:34:46.789161 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7681b45-a09d-4a82-ba56-9df6460a51f1 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:46.789698 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7681b45-a09d-4a82-ba56-9df6460a51f1 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:46.789698 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7681b45-a09d-4a82-ba56-9df6460a51f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:46.808942 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-d7681b45-a09d-4a82-ba56-9df6460a51f1 None None] MFA Rules not processed for user `e1ff3076fbb342caaf7d617c079a9080`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:34:46.810171 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-d7681b45-a09d-4a82-ba56-9df6460a51f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:46.817246 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1218/2423] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:34:46 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:34:47.904992 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=7 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:34:48.770761 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6737e013-4e5f-47d2-bbaa-de84efb121d9 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:48.771119 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6737e013-4e5f-47d2-bbaa-de84efb121d9 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:48.771531 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6737e013-4e5f-47d2-bbaa-de84efb121d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:48.792697 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-6737e013-4e5f-47d2-bbaa-de84efb121d9 None None] MFA Rules not processed for user `7a3f8b5eb09f4ba29f4bd343bd91f6a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:34:48.793948 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-6737e013-4e5f-47d2-bbaa-de84efb121d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:34:48.806735 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1206/2424] 10.222.0.42 () {58 vars in 1009 bytes} [Thu Oct 27 01:34:48 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:34:48.819808 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7681b45-a09d-4a82-ba56-9df6460a51f1 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:34:48.832090 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7681b45-a09d-4a82-ba56-9df6460a51f1 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:34:48.838199 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ed01e6-6130-4084-8747-ca064c31c4af service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77f2cf9e0c3541f9bccfb7e213d88609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:34:48.838850 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ed01e6-6130-4084-8747-ca064c31c4af service swift] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:34:48.839126 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ed01e6-6130-4084-8747-ca064c31c4af service swift] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:34:48.839392 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ed01e6-6130-4084-8747-ca064c31c4af service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:34:48.847413 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ed01e6-6130-4084-8747-ca064c31c4af service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:34:48.848977 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ed01e6-6130-4084-8747-ca064c31c4af service swift] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:34:48.858025 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1219/2425] 10.222.0.42 () {60 vars in 1384 bytes} [Thu Oct 27 01:34:48 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 27 01:34:58.786875 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:34:58.854870 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=26 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:35:02.203216 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6737e013-4e5f-47d2-bbaa-de84efb121d9 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:35:02.204915 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-6737e013-4e5f-47d2-bbaa-de84efb121d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:35:02.291592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6737e013-4e5f-47d2-bbaa-de84efb121d9 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:35:02.297569 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f21a8a4-a40a-4569-a325-1afcfd650f3a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:35:02.298178 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f21a8a4-a40a-4569-a325-1afcfd650f3a service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:35:02.298454 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f21a8a4-a40a-4569-a325-1afcfd650f3a service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:35:02.298720 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f21a8a4-a40a-4569-a325-1afcfd650f3a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:35:02.300476 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-7f21a8a4-a40a-4569-a325-1afcfd650f3a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:35:02.313322 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f21a8a4-a40a-4569-a325-1afcfd650f3a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:35:02.314904 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f21a8a4-a40a-4569-a325-1afcfd650f3a service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:35:02.325870 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1207/2426] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:35:02 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:35:12.321475 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=34 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:35:18.434984 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ed01e6-6130-4084-8747-ca064c31c4af service swift] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:35:18.448415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ed01e6-6130-4084-8747-ca064c31c4af service swift] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:35:18.453892 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a2352b-058d-4c88-aaa6-b3e0e159b770 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:35:18.454491 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a2352b-058d-4c88-aaa6-b3e0e159b770 service nova] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:35:18.454772 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a2352b-058d-4c88-aaa6-b3e0e159b770 service nova] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:35:18.455041 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a2352b-058d-4c88-aaa6-b3e0e159b770 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:35:18.456931 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-20a2352b-058d-4c88-aaa6-b3e0e159b770 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:35:18.469304 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a2352b-058d-4c88-aaa6-b3e0e159b770 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:35:18.470886 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a2352b-058d-4c88-aaa6-b3e0e159b770 service nova] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:35:18.482165 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1220/2427] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:35:18 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:35:28.477893 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=6 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:35:29.737287 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53da93ba-ec0a-4ec0-a7fb-382f4bd4aaba None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:35:29.737826 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53da93ba-ec0a-4ec0-a7fb-382f4bd4aaba None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:35:29.737938 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53da93ba-ec0a-4ec0-a7fb-382f4bd4aaba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:35:29.758680 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-53da93ba-ec0a-4ec0-a7fb-382f4bd4aaba None None] MFA Rules not processed for user `608f1351ab84402d8634ba99b6e26323`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:35:29.759895 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-53da93ba-ec0a-4ec0-a7fb-382f4bd4aaba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:35:29.815319 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1208/2428] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:35:29 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:35:30.115697 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a2352b-058d-4c88-aaa6-b3e0e159b770 service nova] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:35:30.117185 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-20a2352b-058d-4c88-aaa6-b3e0e159b770 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:35:30.173784 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a2352b-058d-4c88-aaa6-b3e0e159b770 service nova] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:35:30.177363 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3d3ddb-758c-4011-b1dc-885e95b1c528 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:35:30.177897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3d3ddb-758c-4011-b1dc-885e95b1c528 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:35:30.178170 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3d3ddb-758c-4011-b1dc-885e95b1c528 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:35:30.178435 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3d3ddb-758c-4011-b1dc-885e95b1c528 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:35:30.180131 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-9d3d3ddb-758c-4011-b1dc-885e95b1c528 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:35:30.189980 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d3d3ddb-758c-4011-b1dc-885e95b1c528 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:35:30.191372 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d3d3ddb-758c-4011-b1dc-885e95b1c528 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:35:30.199061 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1221/2429] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:35:30 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:35:30.512370 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53da93ba-ec0a-4ec0-a7fb-382f4bd4aaba None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:35:30.513902 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-53da93ba-ec0a-4ec0-a7fb-382f4bd4aaba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:35:30.601711 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53da93ba-ec0a-4ec0-a7fb-382f4bd4aaba None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:35:30.606882 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4024eb9-bbbe-47b5-880b-5de36513ab66 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62051562fc0b4cd08750d2293e71787c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:35:30.607477 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4024eb9-bbbe-47b5-880b-5de36513ab66 service placement] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:35:30.607751 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4024eb9-bbbe-47b5-880b-5de36513ab66 service placement] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:35:30.608016 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4024eb9-bbbe-47b5-880b-5de36513ab66 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:35:30.610267 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-c4024eb9-bbbe-47b5-880b-5de36513ab66 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:35:30.621338 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4024eb9-bbbe-47b5-880b-5de36513ab66 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:35:30.622891 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4024eb9-bbbe-47b5-880b-5de36513ab66 service placement] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:35:30.633610 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1209/2430] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:35:30 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:35:34.168540 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3d3ddb-758c-4011-b1dc-885e95b1c528 service neutron] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:35:34.178070 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3d3ddb-758c-4011-b1dc-885e95b1c528 service neutron] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:35:34.181641 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd596d2b-eaf5-48a5-a2cb-24325850b0ed service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:35:34.182234 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd596d2b-eaf5-48a5-a2cb-24325850b0ed service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:35:34.182509 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd596d2b-eaf5-48a5-a2cb-24325850b0ed service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:35:34.182776 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd596d2b-eaf5-48a5-a2cb-24325850b0ed service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:35:34.192505 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd596d2b-eaf5-48a5-a2cb-24325850b0ed service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:35:34.193947 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd596d2b-eaf5-48a5-a2cb-24325850b0ed service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:35:34.203340 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1222/2431] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:35:34 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:35:35.822826 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4024eb9-bbbe-47b5-880b-5de36513ab66 service placement] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:35:35.825504 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-c4024eb9-bbbe-47b5-880b-5de36513ab66 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:35:35.847518 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4024eb9-bbbe-47b5-880b-5de36513ab66 service placement] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:35:35.855159 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d3bcf8-76a0-4469-954e-6072e39f2834 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:35:35.855850 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d3bcf8-76a0-4469-954e-6072e39f2834 service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:35:35.856319 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d3bcf8-76a0-4469-954e-6072e39f2834 service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:35:35.856789 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d3bcf8-76a0-4469-954e-6072e39f2834 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:35:35.858840 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f4d3bcf8-76a0-4469-954e-6072e39f2834 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:35:35.873761 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d3bcf8-76a0-4469-954e-6072e39f2834 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:35:35.875457 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d3bcf8-76a0-4469-954e-6072e39f2834 service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:35:35.909501 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1210/2432] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:35:35 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:35:38.009289 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522219c0-2803-4799-ac0c-ce9b6967ca77 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:35:38.010488 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522219c0-2803-4799-ac0c-ce9b6967ca77 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:35:38.011040 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522219c0-2803-4799-ac0c-ce9b6967ca77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:35:38.038488 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-522219c0-2803-4799-ac0c-ce9b6967ca77 None None] MFA Rules not processed for user `2321134e938247b09711047a12b86382`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:35:38.040378 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-522219c0-2803-4799-ac0c-ce9b6967ca77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:35:38.148172 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1223/2433] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:35:38 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:35:45.892441 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=63 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:35:48.135935 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=57 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:36:43.915124 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d3bcf8-76a0-4469-954e-6072e39f2834 service nova] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:36:43.916578 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f4d3bcf8-76a0-4469-954e-6072e39f2834 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:36:43.984032 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d3bcf8-76a0-4469-954e-6072e39f2834 service nova] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:36:43.988448 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d89102-c213-45d9-9306-931e22eb2ebe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:36:43.990217 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d89102-c213-45d9-9306-931e22eb2ebe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:36:43.990495 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d89102-c213-45d9-9306-931e22eb2ebe admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:36:43.990787 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d89102-c213-45d9-9306-931e22eb2ebe admin admin] PATH_INFO: `/v3/users/2321134e938247b09711047a12b86382` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:36:43.991903 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d89102-c213-45d9-9306-931e22eb2ebe admin admin] RBAC: Authorizing `identity:delete_user(user_id=2321134e938247b09711047a12b86382)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:36:43.993836 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d89102-c213-45d9-9306-931e22eb2ebe admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:36:44.039611 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-40d89102-c213-45d9-9306-931e22eb2ebe admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2321134e938247b09711047a12b86382'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:36:44.046642 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-40d89102-c213-45d9-9306-931e22eb2ebe admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2321134e938247b09711047a12b86382'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:36:44.047266 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-40d89102-c213-45d9-9306-931e22eb2ebe admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2321134e938247b09711047a12b86382'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:36:44.059980 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-40d89102-c213-45d9-9306-931e22eb2ebe admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2321134e938247b09711047a12b86382'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:36:44.066765 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-40d89102-c213-45d9-9306-931e22eb2ebe admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2321134e938247b09711047a12b86382'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:36:44.072622 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1211/2434] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:36:43 2022] DELETE /identity/v3/users/2321134e938247b09711047a12b86382 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:36:44.249028 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522219c0-2803-4799-ac0c-ce9b6967ca77 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:36:44.261176 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522219c0-2803-4799-ac0c-ce9b6967ca77 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:36:44.319606 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f1696f3-aa63-4404-a719-d67b3e2e0734 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:36:44.320226 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1696f3-aa63-4404-a719-d67b3e2e0734 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:36:44.320504 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1696f3-aa63-4404-a719-d67b3e2e0734 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:36:44.320770 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1696f3-aa63-4404-a719-d67b3e2e0734 admin admin] PATH_INFO: `/v3/projects/176e5a3c112e4967a83375c948008ab5` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:36:44.321894 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f1696f3-aa63-4404-a719-d67b3e2e0734 admin admin] RBAC: Authorizing `identity:delete_project(project_id=176e5a3c112e4967a83375c948008ab5)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:36:44.323798 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f1696f3-aa63-4404-a719-d67b3e2e0734 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:36:44.369719 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-6f1696f3-aa63-4404-a719-d67b3e2e0734 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '176e5a3c112e4967a83375c948008ab5', 'request_id': 'req-6f1696f3-aa63-4404-a719-d67b3e2e0734'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:36:44.376901 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-6f1696f3-aa63-4404-a719-d67b3e2e0734 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '176e5a3c112e4967a83375c948008ab5', 'request_id': 'req-6f1696f3-aa63-4404-a719-d67b3e2e0734'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:36:44.383066 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-6f1696f3-aa63-4404-a719-d67b3e2e0734 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '176e5a3c112e4967a83375c948008ab5', 'request_id': 'req-6f1696f3-aa63-4404-a719-d67b3e2e0734'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:36:44.393141 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-6f1696f3-aa63-4404-a719-d67b3e2e0734 admin admin] The token cache is being invalidate because project 176e5a3c112e4967a83375c948008ab5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:36:44.393425 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-6f1696f3-aa63-4404-a719-d67b3e2e0734 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:36:44.394039 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-6f1696f3-aa63-4404-a719-d67b3e2e0734 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:36:44.395992 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1224/2435] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:36:44 2022] DELETE /identity/v3/projects/176e5a3c112e4967a83375c948008ab5 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:36:44.400896 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084f185e-653a-4239-a361-7cb4ecc6f8c8 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:36:44.401221 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084f185e-653a-4239-a361-7cb4ecc6f8c8 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:36:44.401500 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084f185e-653a-4239-a361-7cb4ecc6f8c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:36:44.421351 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-084f185e-653a-4239-a361-7cb4ecc6f8c8 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:36:44.422525 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-084f185e-653a-4239-a361-7cb4ecc6f8c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:36:44.480443 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1212/2436] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:36:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:36:44.488450 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60d74c4-26a4-4811-9509-75aec1f9aff9 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:36:44.488734 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60d74c4-26a4-4811-9509-75aec1f9aff9 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:36:44.489010 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60d74c4-26a4-4811-9509-75aec1f9aff9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:36:44.509278 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-a60d74c4-26a4-4811-9509-75aec1f9aff9 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:36:44.510506 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-a60d74c4-26a4-4811-9509-75aec1f9aff9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:36:44.518786 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1225/2437] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:36:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:36:44.523432 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084f185e-653a-4239-a361-7cb4ecc6f8c8 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:36:44.533083 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084f185e-653a-4239-a361-7cb4ecc6f8c8 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:36:44.538260 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6710baf0-7919-4626-85e4-3d626572203c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:36:44.538853 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6710baf0-7919-4626-85e4-3d626572203c admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:36:44.539133 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6710baf0-7919-4626-85e4-3d626572203c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:36:44.539407 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6710baf0-7919-4626-85e4-3d626572203c admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:36:44.540050 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6710baf0-7919-4626-85e4-3d626572203c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:36:44.540351 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6710baf0-7919-4626-85e4-3d626572203c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:36:44.541974 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6710baf0-7919-4626-85e4-3d626572203c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:36:44.553577 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1213/2438] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:36:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:36:44.557736 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60d74c4-26a4-4811-9509-75aec1f9aff9 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:36:44.568185 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60d74c4-26a4-4811-9509-75aec1f9aff9 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:36:44.572809 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a8e429-37fe-42bc-a2cb-12693656fe2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:36:44.573435 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a8e429-37fe-42bc-a2cb-12693656fe2d admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:36:44.573714 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a8e429-37fe-42bc-a2cb-12693656fe2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:36:44.573985 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a8e429-37fe-42bc-a2cb-12693656fe2d admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:36:44.574606 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a8e429-37fe-42bc-a2cb-12693656fe2d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:36:44.576488 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a8e429-37fe-42bc-a2cb-12693656fe2d admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:36:44.604927 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1226/2439] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:36:44 2022] POST /identity/v3/projects => generated 391 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:36:44.609027 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6710baf0-7919-4626-85e4-3d626572203c admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:36:44.619771 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6710baf0-7919-4626-85e4-3d626572203c admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:36:44.623754 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:36:44.679261 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b9ba035-e0b2-4019-a183-7a1b4e960621 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:36:44.679787 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9ba035-e0b2-4019-a183-7a1b4e960621 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:36:44.680064 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9ba035-e0b2-4019-a183-7a1b4e960621 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:36:44.680336 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9ba035-e0b2-4019-a183-7a1b4e960621 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:36:44.680967 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b9ba035-e0b2-4019-a183-7a1b4e960621 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:36:44.682909 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b9ba035-e0b2-4019-a183-7a1b4e960621 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:36:44.704382 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1214/2440] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:36:44 2022] POST /identity/v3/users => generated 296 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:36:44.708960 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a8e429-37fe-42bc-a2cb-12693656fe2d admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:36:44.719049 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a8e429-37fe-42bc-a2cb-12693656fe2d admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:36:44.724626 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c661a7bd-5198-4729-a0be-9c4582b634ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:36:44.725250 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c661a7bd-5198-4729-a0be-9c4582b634ca admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:36:44.725551 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c661a7bd-5198-4729-a0be-9c4582b634ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:36:44.725817 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c661a7bd-5198-4729-a0be-9c4582b634ca admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:36:44.726471 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c661a7bd-5198-4729-a0be-9c4582b634ca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:36:44.728076 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c661a7bd-5198-4729-a0be-9c4582b634ca admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:36:44.737755 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1227/2441] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:36:44 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:36:44.742371 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b9ba035-e0b2-4019-a183-7a1b4e960621 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:36:44.752149 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b9ba035-e0b2-4019-a183-7a1b4e960621 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:36:44.757120 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd92a917-b8d2-44ba-a11d-b9e919ea2393 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:36:44.757706 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd92a917-b8d2-44ba-a11d-b9e919ea2393 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:36:44.757985 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd92a917-b8d2-44ba-a11d-b9e919ea2393 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:36:44.758258 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd92a917-b8d2-44ba-a11d-b9e919ea2393 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:36:44.758887 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd92a917-b8d2-44ba-a11d-b9e919ea2393 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:36:44.760467 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd92a917-b8d2-44ba-a11d-b9e919ea2393 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:36:44.770929 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1215/2442] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:36:44 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:36:44.774828 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c661a7bd-5198-4729-a0be-9c4582b634ca admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:36:44.784914 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c661a7bd-5198-4729-a0be-9c4582b634ca admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:36:44.790492 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf88c67-b1e7-4154-b58f-37ec30f26c00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:36:44.791124 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf88c67-b1e7-4154-b58f-37ec30f26c00 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:36:44.791395 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf88c67-b1e7-4154-b58f-37ec30f26c00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:36:44.791659 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf88c67-b1e7-4154-b58f-37ec30f26c00 admin admin] PATH_INFO: `/v3/projects/f5795b5470394b9e93d1f68bc267f5bc/users/fe600a11ba1b4ebb9425f844d1d5b4d0/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:36:44.807731 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf88c67-b1e7-4154-b58f-37ec30f26c00 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f5795b5470394b9e93d1f68bc267f5bc, user_id=fe600a11ba1b4ebb9425f844d1d5b4d0, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:36:44.810422 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf88c67-b1e7-4154-b58f-37ec30f26c00 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:36:44.823983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1228/2443] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:36:44 2022] PUT /identity/v3/projects/f5795b5470394b9e93d1f68bc267f5bc/users/fe600a11ba1b4ebb9425f844d1d5b4d0/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:36:44.891965 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e8eae3-fada-4855-a14e-cab1ba9a5689 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:36:44.892238 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e8eae3-fada-4855-a14e-cab1ba9a5689 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:36:44.892505 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e8eae3-fada-4855-a14e-cab1ba9a5689 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:36:44.921277 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-24e8eae3-fada-4855-a14e-cab1ba9a5689 None None] MFA Rules not processed for user `fe600a11ba1b4ebb9425f844d1d5b4d0`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:36:44.922465 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-24e8eae3-fada-4855-a14e-cab1ba9a5689 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:36:44.986295 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1216/2444] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:36:44 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:36:44.997356 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf88c67-b1e7-4154-b58f-37ec30f26c00 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:36:44.999023 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-7bf88c67-b1e7-4154-b58f-37ec30f26c00 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:36:45.030341 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:36:45.059431 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf88c67-b1e7-4154-b58f-37ec30f26c00 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:36:45.063064 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70492dee-79ab-491b-9b9c-b5f66ae57454 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:36:45.063661 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70492dee-79ab-491b-9b9c-b5f66ae57454 service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:36:45.063937 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70492dee-79ab-491b-9b9c-b5f66ae57454 service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:36:45.064214 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70492dee-79ab-491b-9b9c-b5f66ae57454 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:36:45.073239 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70492dee-79ab-491b-9b9c-b5f66ae57454 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:36:45.074717 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70492dee-79ab-491b-9b9c-b5f66ae57454 service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:36:45.083347 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1229/2445] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:36:44 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:36:54.983286 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=63,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:36:55.079393 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=14 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:37:06.423719 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e8eae3-fada-4855-a14e-cab1ba9a5689 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:37:06.485816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e8eae3-fada-4855-a14e-cab1ba9a5689 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:37:06.490355 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6504f3-4ebc-4df5-b70c-fb6b32293acb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:37:06.490993 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6504f3-4ebc-4df5-b70c-fb6b32293acb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:37:06.491271 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6504f3-4ebc-4df5-b70c-fb6b32293acb admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:37:06.491541 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6504f3-4ebc-4df5-b70c-fb6b32293acb admin admin] PATH_INFO: `/v3/users/fe600a11ba1b4ebb9425f844d1d5b4d0` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:37:06.492920 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6504f3-4ebc-4df5-b70c-fb6b32293acb admin admin] RBAC: Authorizing `identity:delete_user(user_id=fe600a11ba1b4ebb9425f844d1d5b4d0)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:37:06.494859 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6504f3-4ebc-4df5-b70c-fb6b32293acb admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:37:06.541832 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-ee6504f3-4ebc-4df5-b70c-fb6b32293acb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fe600a11ba1b4ebb9425f844d1d5b4d0'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:37:06.549161 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-ee6504f3-4ebc-4df5-b70c-fb6b32293acb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fe600a11ba1b4ebb9425f844d1d5b4d0'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:37:06.549785 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-ee6504f3-4ebc-4df5-b70c-fb6b32293acb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fe600a11ba1b4ebb9425f844d1d5b4d0'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:37:06.562698 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-ee6504f3-4ebc-4df5-b70c-fb6b32293acb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fe600a11ba1b4ebb9425f844d1d5b4d0'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:37:06.569404 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-ee6504f3-4ebc-4df5-b70c-fb6b32293acb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fe600a11ba1b4ebb9425f844d1d5b4d0'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:37:06.576430 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1217/2446] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:37:06 2022] DELETE /identity/v3/users/fe600a11ba1b4ebb9425f844d1d5b4d0 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:37:06.580800 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7437e51-1e92-48fb-b462-851dab32188a None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:37:06.581314 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7437e51-1e92-48fb-b462-851dab32188a None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:37:06.581431 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7437e51-1e92-48fb-b462-851dab32188a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:37:06.600934 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-c7437e51-1e92-48fb-b462-851dab32188a None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:37:06.602679 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-c7437e51-1e92-48fb-b462-851dab32188a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:37:06.661033 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1230/2447] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:37:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:37:06.671393 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6504f3-4ebc-4df5-b70c-fb6b32293acb admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:37:06.672810 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ee6504f3-4ebc-4df5-b70c-fb6b32293acb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:37:06.728894 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6504f3-4ebc-4df5-b70c-fb6b32293acb admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:37:06.732455 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8033f0-17eb-469f-8a12-9ac6da846963 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:37:06.733077 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8033f0-17eb-469f-8a12-9ac6da846963 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:37:06.733437 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8033f0-17eb-469f-8a12-9ac6da846963 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:37:06.733706 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8033f0-17eb-469f-8a12-9ac6da846963 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:37:06.744228 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d8033f0-17eb-469f-8a12-9ac6da846963 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:37:06.745690 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d8033f0-17eb-469f-8a12-9ac6da846963 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:37:06.755095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1218/2448] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:37:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:37:07.024044 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7437e51-1e92-48fb-b462-851dab32188a None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:37:07.025564 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-c7437e51-1e92-48fb-b462-851dab32188a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:37:07.037626 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7437e51-1e92-48fb-b462-851dab32188a None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:37:07.043586 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c7cb805-75b8-494e-9a5a-3b9b73c56790 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:37:07.044216 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7cb805-75b8-494e-9a5a-3b9b73c56790 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:37:07.044611 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7cb805-75b8-494e-9a5a-3b9b73c56790 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:37:07.044882 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7cb805-75b8-494e-9a5a-3b9b73c56790 admin admin] PATH_INFO: `/v3/projects/f5795b5470394b9e93d1f68bc267f5bc` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:37:07.046038 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c7cb805-75b8-494e-9a5a-3b9b73c56790 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f5795b5470394b9e93d1f68bc267f5bc)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:37:07.047921 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c7cb805-75b8-494e-9a5a-3b9b73c56790 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:37:07.096828 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-8c7cb805-75b8-494e-9a5a-3b9b73c56790 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f5795b5470394b9e93d1f68bc267f5bc', 'request_id': 'req-8c7cb805-75b8-494e-9a5a-3b9b73c56790'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:37:07.104758 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-8c7cb805-75b8-494e-9a5a-3b9b73c56790 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f5795b5470394b9e93d1f68bc267f5bc', 'request_id': 'req-8c7cb805-75b8-494e-9a5a-3b9b73c56790'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:37:07.111189 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-8c7cb805-75b8-494e-9a5a-3b9b73c56790 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f5795b5470394b9e93d1f68bc267f5bc', 'request_id': 'req-8c7cb805-75b8-494e-9a5a-3b9b73c56790'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:37:07.122085 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-8c7cb805-75b8-494e-9a5a-3b9b73c56790 admin admin] The token cache is being invalidate because project f5795b5470394b9e93d1f68bc267f5bc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:37:07.122372 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-8c7cb805-75b8-494e-9a5a-3b9b73c56790 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:37:07.122999 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-8c7cb805-75b8-494e-9a5a-3b9b73c56790 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:37:07.124472 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1231/2449] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:37:07 2022] DELETE /identity/v3/projects/f5795b5470394b9e93d1f68bc267f5bc => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:37:16.750882 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=79,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:37:17.119110 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=54,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:37:32.328488 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8033f0-17eb-469f-8a12-9ac6da846963 service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:37:32.329988 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-3d8033f0-17eb-469f-8a12-9ac6da846963 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:37:32.402447 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8033f0-17eb-469f-8a12-9ac6da846963 service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:37:32.463594 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc102e2-1a29-429a-b5ce-1874e2d9ea8b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:37:32.464189 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc102e2-1a29-429a-b5ce-1874e2d9ea8b service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:37:32.464466 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc102e2-1a29-429a-b5ce-1874e2d9ea8b service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:37:32.464734 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc102e2-1a29-429a-b5ce-1874e2d9ea8b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:37:32.466482 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-afc102e2-1a29-429a-b5ce-1874e2d9ea8b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:37:32.522596 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afc102e2-1a29-429a-b5ce-1874e2d9ea8b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:37:32.524927 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afc102e2-1a29-429a-b5ce-1874e2d9ea8b service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:37:32.533716 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1219/2450] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:37:32 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 206 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:37:42.529625 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=84 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:37:51.105497 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7133b3a-6ee0-462a-925b-ce29795d79fa None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:37:51.106020 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7133b3a-6ee0-462a-925b-ce29795d79fa None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:37:51.106020 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7133b3a-6ee0-462a-925b-ce29795d79fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:37:51.126330 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-c7133b3a-6ee0-462a-925b-ce29795d79fa None None] MFA Rules not processed for user `7a3f8b5eb09f4ba29f4bd343bd91f6a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:37:51.127528 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-c7133b3a-6ee0-462a-925b-ce29795d79fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:37:51.184963 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1232/2451] 10.222.0.42 () {58 vars in 1009 bytes} [Thu Oct 27 01:37:51 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:37:51.227218 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc102e2-1a29-429a-b5ce-1874e2d9ea8b service nova] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:37:51.228621 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-afc102e2-1a29-429a-b5ce-1874e2d9ea8b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:37:51.299705 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc102e2-1a29-429a-b5ce-1874e2d9ea8b service nova] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:37:51.305092 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c176ba7-95a6-4e38-8d07-998d019631ba service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77f2cf9e0c3541f9bccfb7e213d88609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:37:51.305729 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c176ba7-95a6-4e38-8d07-998d019631ba service swift] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:37:51.306007 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c176ba7-95a6-4e38-8d07-998d019631ba service swift] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:37:51.306280 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c176ba7-95a6-4e38-8d07-998d019631ba service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:37:51.314358 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c176ba7-95a6-4e38-8d07-998d019631ba service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:37:51.315938 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c176ba7-95a6-4e38-8d07-998d019631ba service swift] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:37:51.323883 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1220/2452] 10.222.0.42 () {60 vars in 1384 bytes} [Thu Oct 27 01:37:51 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 27 01:38:01.177912 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=25 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:38:01.320710 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=32 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:38:03.380504 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7133b3a-6ee0-462a-925b-ce29795d79fa None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:38:03.381937 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-c7133b3a-6ee0-462a-925b-ce29795d79fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:38:03.449754 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7133b3a-6ee0-462a-925b-ce29795d79fa None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:38:03.454549 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e969581-8059-4966-afa5-9c179c3e26c5 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62051562fc0b4cd08750d2293e71787c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:38:03.455144 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e969581-8059-4966-afa5-9c179c3e26c5 service placement] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:38:03.455419 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e969581-8059-4966-afa5-9c179c3e26c5 service placement] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:38:03.455688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e969581-8059-4966-afa5-9c179c3e26c5 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:38:03.457378 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-8e969581-8059-4966-afa5-9c179c3e26c5 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:38:03.468381 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e969581-8059-4966-afa5-9c179c3e26c5 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:38:03.470027 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e969581-8059-4966-afa5-9c179c3e26c5 service placement] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:38:03.480376 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1233/2453] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:38:03 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:38:05.380089 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc336674-0a37-4960-bc84-b6f8375ac6c0 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:38:05.380644 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc336674-0a37-4960-bc84-b6f8375ac6c0 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:38:05.380644 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc336674-0a37-4960-bc84-b6f8375ac6c0 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:38:05.381991 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1221/2454] 10.222.0.42 () {54 vars in 921 bytes} [Thu Oct 27 01:38:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:38:05.389328 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e31830-795f-40bc-9abc-789f9eeae0c6 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:38:05.389830 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e31830-795f-40bc-9abc-789f9eeae0c6 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:38:05.389830 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e31830-795f-40bc-9abc-789f9eeae0c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:38:05.429440 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-a7e31830-795f-40bc-9abc-789f9eeae0c6 None None] MFA Rules not processed for user `51033d56f85f4282a49ded6338440677`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:38:05.430718 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-a7e31830-795f-40bc-9abc-789f9eeae0c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:38:05.439883 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1234/2455] 10.222.0.42 () {58 vars in 1018 bytes} [Thu Oct 27 01:38:05 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:38:05.451377 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc336674-0a37-4960-bc84-b6f8375ac6c0 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:38:05.453099 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-fc336674-0a37-4960-bc84-b6f8375ac6c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:38:05.541518 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc336674-0a37-4960-bc84-b6f8375ac6c0 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:38:05.545129 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e5c990-8f14-4961-956f-28f4d27b6c19 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2880dbb00eec471285d242e8183dd8d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:38:05.545731 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e5c990-8f14-4961-956f-28f4d27b6c19 service glance] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:38:05.546009 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e5c990-8f14-4961-956f-28f4d27b6c19 service glance] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:38:05.546280 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e5c990-8f14-4961-956f-28f4d27b6c19 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:38:05.557088 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e5c990-8f14-4961-956f-28f4d27b6c19 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:38:05.558525 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e5c990-8f14-4961-956f-28f4d27b6c19 service glance] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:38:05.569120 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1222/2456] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:38:05 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:38:05.637843 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb9dd5a-eace-4f02-af52-a49783b4626f None None] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:38:05.638132 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb9dd5a-eace-4f02-af52-a49783b4626f None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:38:05.638406 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb9dd5a-eace-4f02-af52-a49783b4626f None None] PATH_INFO: `/` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:38:05.639753 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1235/2457] 10.222.0.42 () {54 vars in 921 bytes} [Thu Oct 27 01:38:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:38:05.644440 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68ece32-1f1d-411a-b513-d6f88e76c17d None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:38:05.644727 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68ece32-1f1d-411a-b513-d6f88e76c17d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:38:05.645032 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68ece32-1f1d-411a-b513-d6f88e76c17d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:38:05.684271 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-e68ece32-1f1d-411a-b513-d6f88e76c17d None None] MFA Rules not processed for user `4072417990c94b0488871332d28224ba`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:38:05.685444 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e68ece32-1f1d-411a-b513-d6f88e76c17d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:38:05.734983 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1223/2458] 10.222.0.42 () {58 vars in 1018 bytes} [Thu Oct 27 01:38:05 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:38:05.748335 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb9dd5a-eace-4f02-af52-a49783b4626f None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:38:05.750044 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-fcb9dd5a-eace-4f02-af52-a49783b4626f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:38:05.763469 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb9dd5a-eace-4f02-af52-a49783b4626f None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:38:05.766952 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f710bf9d-ad1a-4e7c-8e9a-38801e85c914 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:38:05.767532 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f710bf9d-ad1a-4e7c-8e9a-38801e85c914 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:38:05.767809 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f710bf9d-ad1a-4e7c-8e9a-38801e85c914 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:38:05.768079 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f710bf9d-ad1a-4e7c-8e9a-38801e85c914 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:38:05.775381 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f710bf9d-ad1a-4e7c-8e9a-38801e85c914 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:38:05.776788 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f710bf9d-ad1a-4e7c-8e9a-38801e85c914 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:38:05.784344 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1236/2459] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:38:05 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:38:05.872316 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5417ba2f-4d32-4402-b385-68171a37e662 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:38:05.872593 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5417ba2f-4d32-4402-b385-68171a37e662 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:38:05.872861 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5417ba2f-4d32-4402-b385-68171a37e662 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:38:05.874270 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1224/2460] 10.222.0.42 () {54 vars in 921 bytes} [Thu Oct 27 01:38:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 27 01:38:05.879517 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e992a81-e351-4054-8bf7-50b5028c990f None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:38:05.879799 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e992a81-e351-4054-8bf7-50b5028c990f None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:38:05.880072 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e992a81-e351-4054-8bf7-50b5028c990f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:38:05.899701 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-3e992a81-e351-4054-8bf7-50b5028c990f None None] MFA Rules not processed for user `51033d56f85f4282a49ded6338440677`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:38:05.900883 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-3e992a81-e351-4054-8bf7-50b5028c990f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:38:05.910001 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1237/2461] 10.222.0.42 () {58 vars in 1018 bytes} [Thu Oct 27 01:38:05 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:38:05.921015 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5417ba2f-4d32-4402-b385-68171a37e662 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:38:05.933295 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5417ba2f-4d32-4402-b385-68171a37e662 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:38:05.937015 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4402da02-3aa3-487d-b812-fb6209f6c241 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:38:05.937592 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4402da02-3aa3-487d-b812-fb6209f6c241 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:38:05.937866 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4402da02-3aa3-487d-b812-fb6209f6c241 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:38:05.938136 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4402da02-3aa3-487d-b812-fb6209f6c241 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:38:05.949338 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4402da02-3aa3-487d-b812-fb6209f6c241 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:38:05.950760 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4402da02-3aa3-487d-b812-fb6209f6c241 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:38:05.961206 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1225/2462] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:38:05 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:38:07.747835 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d89e8df-c272-4dee-91fc-b73e19fbf597 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:38:07.748315 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d89e8df-c272-4dee-91fc-b73e19fbf597 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:38:07.748410 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d89e8df-c272-4dee-91fc-b73e19fbf597 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:38:07.768146 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-3d89e8df-c272-4dee-91fc-b73e19fbf597 None None] MFA Rules not processed for user `7a3f8b5eb09f4ba29f4bd343bd91f6a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:38:07.769507 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-3d89e8df-c272-4dee-91fc-b73e19fbf597 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:38:07.776662 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1238/2463] 10.222.0.42 () {58 vars in 1009 bytes} [Thu Oct 27 01:38:07 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:38:07.788409 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4402da02-3aa3-487d-b812-fb6209f6c241 service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:38:07.799748 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4402da02-3aa3-487d-b812-fb6209f6c241 service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:38:07.805140 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9d039c-0d97-4feb-8610-f165f891bc11 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77f2cf9e0c3541f9bccfb7e213d88609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:38:07.806279 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9d039c-0d97-4feb-8610-f165f891bc11 service swift] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:38:07.806553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9d039c-0d97-4feb-8610-f165f891bc11 service swift] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:38:07.806830 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9d039c-0d97-4feb-8610-f165f891bc11 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:38:07.815432 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c9d039c-0d97-4feb-8610-f165f891bc11 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:38:07.817437 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c9d039c-0d97-4feb-8610-f165f891bc11 service swift] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:38:07.826211 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1226/2464] 10.222.0.42 () {60 vars in 1384 bytes} [Thu Oct 27 01:38:07 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 27 01:38:17.762630 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=53 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:38:17.823354 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=76 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:38:26.709826 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f666815f-1f52-4fe2-a2eb-916922bfd7b0 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:38:26.710209 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f666815f-1f52-4fe2-a2eb-916922bfd7b0 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:38:26.710392 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f666815f-1f52-4fe2-a2eb-916922bfd7b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:38:26.730038 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-f666815f-1f52-4fe2-a2eb-916922bfd7b0 None None] MFA Rules not processed for user `7a3f8b5eb09f4ba29f4bd343bd91f6a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:38:26.731277 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-f666815f-1f52-4fe2-a2eb-916922bfd7b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:38:26.738459 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1239/2465] 10.222.0.42 () {58 vars in 967 bytes} [Thu Oct 27 01:38:26 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:38:26.752064 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9d039c-0d97-4feb-8610-f165f891bc11 service swift] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:38:26.762653 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9d039c-0d97-4feb-8610-f165f891bc11 service swift] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:38:26.767565 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e777c03-8627-49ef-9237-ceecf8ac1168 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77f2cf9e0c3541f9bccfb7e213d88609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:38:26.768153 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e777c03-8627-49ef-9237-ceecf8ac1168 service swift] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:38:26.768425 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e777c03-8627-49ef-9237-ceecf8ac1168 service swift] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:38:26.768690 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e777c03-8627-49ef-9237-ceecf8ac1168 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:38:26.777137 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e777c03-8627-49ef-9237-ceecf8ac1168 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:38:26.778708 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e777c03-8627-49ef-9237-ceecf8ac1168 service swift] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:38:26.786844 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1227/2466] 10.222.0.42 () {60 vars in 1384 bytes} [Thu Oct 27 01:38:26 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 27 01:38:36.724116 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=3 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:38:36.783589 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=6 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:38:41.058567 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1dc4f4-bb6e-4c62-b1d6-d1e0f07222d8 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:38:41.059041 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1dc4f4-bb6e-4c62-b1d6-d1e0f07222d8 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:38:41.059141 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1dc4f4-bb6e-4c62-b1d6-d1e0f07222d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:38:41.078137 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-9c1dc4f4-bb6e-4c62-b1d6-d1e0f07222d8 None None] MFA Rules not processed for user `7a3f8b5eb09f4ba29f4bd343bd91f6a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:38:41.079322 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-9c1dc4f4-bb6e-4c62-b1d6-d1e0f07222d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:38:41.086567 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1240/2467] 10.222.0.42 () {58 vars in 967 bytes} [Thu Oct 27 01:38:41 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:38:41.099125 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e777c03-8627-49ef-9237-ceecf8ac1168 service swift] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:38:41.110206 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e777c03-8627-49ef-9237-ceecf8ac1168 service swift] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:38:41.115379 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f4bab3-717d-4033-8798-00e116bf913e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77f2cf9e0c3541f9bccfb7e213d88609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:38:41.115966 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f4bab3-717d-4033-8798-00e116bf913e service swift] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:38:41.116246 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f4bab3-717d-4033-8798-00e116bf913e service swift] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:38:41.116518 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f4bab3-717d-4033-8798-00e116bf913e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:38:41.124563 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2f4bab3-717d-4033-8798-00e116bf913e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:38:41.126162 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2f4bab3-717d-4033-8798-00e116bf913e service swift] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:38:41.134234 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1228/2468] 10.222.0.42 () {60 vars in 1384 bytes} [Thu Oct 27 01:38:41 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 27 01:38:44.595681 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1dc4f4-bb6e-4c62-b1d6-d1e0f07222d8 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:38:44.612216 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1dc4f4-bb6e-4c62-b1d6-d1e0f07222d8 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:38:44.617968 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc3dd99-514d-453b-bcf1-f795db50a8ab service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62051562fc0b4cd08750d2293e71787c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:38:44.618693 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc3dd99-514d-453b-bcf1-f795db50a8ab service placement] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:38:44.619049 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc3dd99-514d-453b-bcf1-f795db50a8ab service placement] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:38:44.619432 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc3dd99-514d-453b-bcf1-f795db50a8ab service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:38:44.621800 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-fdc3dd99-514d-453b-bcf1-f795db50a8ab service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:38:44.635533 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdc3dd99-514d-453b-bcf1-f795db50a8ab service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:38:44.637720 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdc3dd99-514d-453b-bcf1-f795db50a8ab service placement] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:38:44.664686 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1241/2469] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:38:44 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 67 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:38:47.867909 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f4bab3-717d-4033-8798-00e116bf913e service swift] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:38:47.879547 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f4bab3-717d-4033-8798-00e116bf913e service swift] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:38:47.884798 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01cf2478-55b2-4c9e-938c-d4a60c816fa3 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62051562fc0b4cd08750d2293e71787c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:38:47.886212 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cf2478-55b2-4c9e-938c-d4a60c816fa3 service placement] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:38:47.886491 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cf2478-55b2-4c9e-938c-d4a60c816fa3 service placement] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:38:47.886758 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cf2478-55b2-4c9e-938c-d4a60c816fa3 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:38:47.888534 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-01cf2478-55b2-4c9e-938c-d4a60c816fa3 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:38:47.900208 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01cf2478-55b2-4c9e-938c-d4a60c816fa3 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:38:47.902428 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01cf2478-55b2-4c9e-938c-d4a60c816fa3 service placement] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:38:47.912717 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1229/2470] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:38:47 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:38:47.933687 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc3dd99-514d-453b-bcf1-f795db50a8ab service placement] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:38:47.935494 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-fdc3dd99-514d-453b-bcf1-f795db50a8ab service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:38:48.006963 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc3dd99-514d-453b-bcf1-f795db50a8ab service placement] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:38:48.012065 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eff5fa4-b381-4900-a473-7e88e399c20f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:38:48.012674 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eff5fa4-b381-4900-a473-7e88e399c20f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:38:48.012949 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eff5fa4-b381-4900-a473-7e88e399c20f admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:38:48.013246 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eff5fa4-b381-4900-a473-7e88e399c20f admin admin] PATH_INFO: `/v3/users/e1ff3076fbb342caaf7d617c079a9080` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:38:48.014344 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eff5fa4-b381-4900-a473-7e88e399c20f admin admin] RBAC: Authorizing `identity:delete_user(user_id=e1ff3076fbb342caaf7d617c079a9080)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:38:48.016246 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eff5fa4-b381-4900-a473-7e88e399c20f admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:38:48.063061 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-8eff5fa4-b381-4900-a473-7e88e399c20f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e1ff3076fbb342caaf7d617c079a9080'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:38:48.071730 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-8eff5fa4-b381-4900-a473-7e88e399c20f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e1ff3076fbb342caaf7d617c079a9080'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:38:48.084659 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-8eff5fa4-b381-4900-a473-7e88e399c20f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e1ff3076fbb342caaf7d617c079a9080'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:38:48.085335 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-8eff5fa4-b381-4900-a473-7e88e399c20f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e1ff3076fbb342caaf7d617c079a9080'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:38:48.085942 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-8eff5fa4-b381-4900-a473-7e88e399c20f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e1ff3076fbb342caaf7d617c079a9080'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:38:48.097830 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1242/2471] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:38:47 2022] DELETE /identity/v3/users/e1ff3076fbb342caaf7d617c079a9080 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:38:48.274408 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01cf2478-55b2-4c9e-938c-d4a60c816fa3 service placement] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:38:48.286485 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01cf2478-55b2-4c9e-938c-d4a60c816fa3 service placement] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:38:48.345823 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3178f46-2465-4703-a401-a0633b1fe94f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:38:48.346379 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3178f46-2465-4703-a401-a0633b1fe94f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:38:48.346661 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3178f46-2465-4703-a401-a0633b1fe94f admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:38:48.346937 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3178f46-2465-4703-a401-a0633b1fe94f admin admin] PATH_INFO: `/v3/projects/9e496297038149beb8bfe69c53234988` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:38:48.348047 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3178f46-2465-4703-a401-a0633b1fe94f admin admin] RBAC: Authorizing `identity:delete_project(project_id=9e496297038149beb8bfe69c53234988)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:38:48.349994 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3178f46-2465-4703-a401-a0633b1fe94f admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:38:48.395301 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d3178f46-2465-4703-a401-a0633b1fe94f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9e496297038149beb8bfe69c53234988', 'request_id': 'req-d3178f46-2465-4703-a401-a0633b1fe94f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:38:48.402594 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d3178f46-2465-4703-a401-a0633b1fe94f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9e496297038149beb8bfe69c53234988', 'request_id': 'req-d3178f46-2465-4703-a401-a0633b1fe94f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:38:48.407951 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d3178f46-2465-4703-a401-a0633b1fe94f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9e496297038149beb8bfe69c53234988', 'request_id': 'req-d3178f46-2465-4703-a401-a0633b1fe94f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:38:48.419615 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d3178f46-2465-4703-a401-a0633b1fe94f admin admin] The token cache is being invalidate because project 9e496297038149beb8bfe69c53234988 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:38:48.419897 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d3178f46-2465-4703-a401-a0633b1fe94f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:38:48.420518 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d3178f46-2465-4703-a401-a0633b1fe94f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:38:48.422010 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1230/2472] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:38:48 2022] DELETE /identity/v3/projects/9e496297038149beb8bfe69c53234988 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:38:48.426523 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eff5fa4-b381-4900-a473-7e88e399c20f admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:38:48.427912 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-8eff5fa4-b381-4900-a473-7e88e399c20f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:38:48.441428 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eff5fa4-b381-4900-a473-7e88e399c20f admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:38:48.497698 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f8d862-cbb6-4c48-86cd-a72bb546b974 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:38:48.498315 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f8d862-cbb6-4c48-86cd-a72bb546b974 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:38:48.498591 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f8d862-cbb6-4c48-86cd-a72bb546b974 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:38:48.498859 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f8d862-cbb6-4c48-86cd-a72bb546b974 admin admin] PATH_INFO: `/v3/users/12e68c094a8f4814a06d3a6143746982` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:38:48.499969 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f8d862-cbb6-4c48-86cd-a72bb546b974 admin admin] RBAC: Authorizing `identity:delete_user(user_id=12e68c094a8f4814a06d3a6143746982)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:38:48.501906 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f8d862-cbb6-4c48-86cd-a72bb546b974 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:38:48.503328 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:38:48.548857 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-77f8d862-cbb6-4c48-86cd-a72bb546b974 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '12e68c094a8f4814a06d3a6143746982'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:38:48.556344 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-77f8d862-cbb6-4c48-86cd-a72bb546b974 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '12e68c094a8f4814a06d3a6143746982'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:38:48.568873 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-77f8d862-cbb6-4c48-86cd-a72bb546b974 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '12e68c094a8f4814a06d3a6143746982'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:38:48.569530 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-77f8d862-cbb6-4c48-86cd-a72bb546b974 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '12e68c094a8f4814a06d3a6143746982'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:38:48.570125 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-77f8d862-cbb6-4c48-86cd-a72bb546b974 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '12e68c094a8f4814a06d3a6143746982'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:38:48.582344 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1243/2473] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:38:48 2022] DELETE /identity/v3/users/12e68c094a8f4814a06d3a6143746982 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:38:48.754553 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3178f46-2465-4703-a401-a0633b1fe94f admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:38:48.756064 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d3178f46-2465-4703-a401-a0633b1fe94f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:38:48.772984 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3178f46-2465-4703-a401-a0633b1fe94f admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:38:48.832891 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1831be-f645-4d9a-bff4-d13be1a84995 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:38:48.833545 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1831be-f645-4d9a-bff4-d13be1a84995 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:38:48.833818 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1831be-f645-4d9a-bff4-d13be1a84995 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:38:48.834086 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1831be-f645-4d9a-bff4-d13be1a84995 admin admin] PATH_INFO: `/v3/projects/3d3b0ef54598449e9b3430810d6a081e` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:38:48.835185 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1831be-f645-4d9a-bff4-d13be1a84995 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3d3b0ef54598449e9b3430810d6a081e)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:38:48.837096 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1831be-f645-4d9a-bff4-d13be1a84995 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:38:48.843766 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:38:48.883607 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-ed1831be-f645-4d9a-bff4-d13be1a84995 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3d3b0ef54598449e9b3430810d6a081e', 'request_id': 'req-ed1831be-f645-4d9a-bff4-d13be1a84995'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:38:48.891670 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-ed1831be-f645-4d9a-bff4-d13be1a84995 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3d3b0ef54598449e9b3430810d6a081e', 'request_id': 'req-ed1831be-f645-4d9a-bff4-d13be1a84995'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:38:48.897092 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-ed1831be-f645-4d9a-bff4-d13be1a84995 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3d3b0ef54598449e9b3430810d6a081e', 'request_id': 'req-ed1831be-f645-4d9a-bff4-d13be1a84995'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:38:48.908574 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-ed1831be-f645-4d9a-bff4-d13be1a84995 admin admin] The token cache is being invalidate because project 3d3b0ef54598449e9b3430810d6a081e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:38:48.908865 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-ed1831be-f645-4d9a-bff4-d13be1a84995 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:38:48.909541 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-ed1831be-f645-4d9a-bff4-d13be1a84995 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:38:48.911035 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1231/2474] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:38:48 2022] DELETE /identity/v3/projects/3d3b0ef54598449e9b3430810d6a081e => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:38:58.578824 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=26,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:38:58.905607 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=22,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:39:23.750968 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f8d862-cbb6-4c48-86cd-a72bb546b974 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:23.752415 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-77f8d862-cbb6-4c48-86cd-a72bb546b974 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:39:23.812951 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f8d862-cbb6-4c48-86cd-a72bb546b974 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:23.863656 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68e2b5c-f490-455e-b71f-b029b05a3a4d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:23.864259 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68e2b5c-f490-455e-b71f-b029b05a3a4d service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:23.864529 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68e2b5c-f490-455e-b71f-b029b05a3a4d service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:23.864817 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68e2b5c-f490-455e-b71f-b029b05a3a4d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:23.866563 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-b68e2b5c-f490-455e-b71f-b029b05a3a4d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:39:23.937174 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68e2b5c-f490-455e-b71f-b029b05a3a4d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:23.938676 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68e2b5c-f490-455e-b71f-b029b05a3a4d service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:23.949550 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1244/2475] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:39:23 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 199 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:39:25.132434 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1831be-f645-4d9a-bff4-d13be1a84995 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:25.133975 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ed1831be-f645-4d9a-bff4-d13be1a84995 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:39:25.146683 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1831be-f645-4d9a-bff4-d13be1a84995 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:25.150334 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff38d3c2-94a5-43d1-964a-2d643291879e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:25.150961 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff38d3c2-94a5-43d1-964a-2d643291879e service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:25.151236 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff38d3c2-94a5-43d1-964a-2d643291879e service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:25.151506 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff38d3c2-94a5-43d1-964a-2d643291879e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:25.153219 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ff38d3c2-94a5-43d1-964a-2d643291879e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:39:25.167265 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff38d3c2-94a5-43d1-964a-2d643291879e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:25.168682 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff38d3c2-94a5-43d1-964a-2d643291879e service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:25.179631 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1232/2476] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:39:25 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:39:30.547832 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68e2b5c-f490-455e-b71f-b029b05a3a4d service neutron] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:30.549268 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-b68e2b5c-f490-455e-b71f-b029b05a3a4d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:39:30.559997 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68e2b5c-f490-455e-b71f-b029b05a3a4d service neutron] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:30.564305 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8a016e-e241-412d-81b4-cd17355eb3f7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:30.564863 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8a016e-e241-412d-81b4-cd17355eb3f7 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:30.565178 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8a016e-e241-412d-81b4-cd17355eb3f7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:30.565451 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8a016e-e241-412d-81b4-cd17355eb3f7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:30.567125 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-1f8a016e-e241-412d-81b4-cd17355eb3f7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:39:30.618015 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:39:30.629545 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f8a016e-e241-412d-81b4-cd17355eb3f7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:30.630959 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f8a016e-e241-412d-81b4-cd17355eb3f7 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:30.640440 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1245/2477] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:39:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:39:33.520225 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff38d3c2-94a5-43d1-964a-2d643291879e service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:33.521765 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ff38d3c2-94a5-43d1-964a-2d643291879e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:39:33.535762 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff38d3c2-94a5-43d1-964a-2d643291879e service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:33.540185 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4b618c-a095-47e9-8ec0-04711c9ca064 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:33.540794 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4b618c-a095-47e9-8ec0-04711c9ca064 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:33.541107 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4b618c-a095-47e9-8ec0-04711c9ca064 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:33.541360 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4b618c-a095-47e9-8ec0-04711c9ca064 admin admin] PATH_INFO: `/v3/users/608f1351ab84402d8634ba99b6e26323` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:33.542462 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff4b618c-a095-47e9-8ec0-04711c9ca064 admin admin] RBAC: Authorizing `identity:delete_user(user_id=608f1351ab84402d8634ba99b6e26323)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:33.544358 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff4b618c-a095-47e9-8ec0-04711c9ca064 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:33.596710 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-ff4b618c-a095-47e9-8ec0-04711c9ca064 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '608f1351ab84402d8634ba99b6e26323'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:33.604419 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-ff4b618c-a095-47e9-8ec0-04711c9ca064 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '608f1351ab84402d8634ba99b6e26323'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:33.605421 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-ff4b618c-a095-47e9-8ec0-04711c9ca064 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '608f1351ab84402d8634ba99b6e26323'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:33.618691 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-ff4b618c-a095-47e9-8ec0-04711c9ca064 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '608f1351ab84402d8634ba99b6e26323'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:33.626580 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-ff4b618c-a095-47e9-8ec0-04711c9ca064 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '608f1351ab84402d8634ba99b6e26323'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:33.632412 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1233/2478] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:39:33 2022] DELETE /identity/v3/users/608f1351ab84402d8634ba99b6e26323 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:39:33.815344 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8a016e-e241-412d-81b4-cd17355eb3f7 service neutron] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:33.825999 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8a016e-e241-412d-81b4-cd17355eb3f7 service neutron] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:33.884598 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980d0bb3-bf52-4dfd-993e-8633ee813188 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:33.885259 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980d0bb3-bf52-4dfd-993e-8633ee813188 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:33.885535 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980d0bb3-bf52-4dfd-993e-8633ee813188 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:33.885803 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980d0bb3-bf52-4dfd-993e-8633ee813188 admin admin] PATH_INFO: `/v3/projects/73c30908c04242e28297b339bfcc5600` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:33.886897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-980d0bb3-bf52-4dfd-993e-8633ee813188 admin admin] RBAC: Authorizing `identity:delete_project(project_id=73c30908c04242e28297b339bfcc5600)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:33.888786 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-980d0bb3-bf52-4dfd-993e-8633ee813188 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:33.936790 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-980d0bb3-bf52-4dfd-993e-8633ee813188 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '73c30908c04242e28297b339bfcc5600', 'request_id': 'req-980d0bb3-bf52-4dfd-993e-8633ee813188'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:33.944166 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-980d0bb3-bf52-4dfd-993e-8633ee813188 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '73c30908c04242e28297b339bfcc5600', 'request_id': 'req-980d0bb3-bf52-4dfd-993e-8633ee813188'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:33.950327 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-980d0bb3-bf52-4dfd-993e-8633ee813188 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '73c30908c04242e28297b339bfcc5600', 'request_id': 'req-980d0bb3-bf52-4dfd-993e-8633ee813188'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:33.960843 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-980d0bb3-bf52-4dfd-993e-8633ee813188 admin admin] The token cache is being invalidate because project 73c30908c04242e28297b339bfcc5600 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:39:33.961156 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-980d0bb3-bf52-4dfd-993e-8633ee813188 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:33.961772 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-980d0bb3-bf52-4dfd-993e-8633ee813188 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:33.963220 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1246/2479] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:39:33 2022] DELETE /identity/v3/projects/73c30908c04242e28297b339bfcc5600 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:39:33.968788 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd357d3a-6c10-431c-8a78-f30189accc49 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:33.969150 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd357d3a-6c10-431c-8a78-f30189accc49 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:33.969377 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd357d3a-6c10-431c-8a78-f30189accc49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:33.988800 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-bd357d3a-6c10-431c-8a78-f30189accc49 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:39:33.990072 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-bd357d3a-6c10-431c-8a78-f30189accc49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:39:34.048288 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1234/2480] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:39:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:39:34.057345 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988917e5-5c09-4fb4-803e-5a592e47d7ae None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:34.057634 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988917e5-5c09-4fb4-803e-5a592e47d7ae None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:34.057954 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988917e5-5c09-4fb4-803e-5a592e47d7ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:34.078386 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-988917e5-5c09-4fb4-803e-5a592e47d7ae None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:39:34.079576 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-988917e5-5c09-4fb4-803e-5a592e47d7ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:39:34.087459 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1247/2481] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:39:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:39:34.091603 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd357d3a-6c10-431c-8a78-f30189accc49 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:34.101868 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd357d3a-6c10-431c-8a78-f30189accc49 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:34.106681 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c187564-de83-4f76-a5fe-28a5d0d18374 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:34.107275 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c187564-de83-4f76-a5fe-28a5d0d18374 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:34.107550 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c187564-de83-4f76-a5fe-28a5d0d18374 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:34.107817 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c187564-de83-4f76-a5fe-28a5d0d18374 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:34.108537 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c187564-de83-4f76-a5fe-28a5d0d18374 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:39:34.108748 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c187564-de83-4f76-a5fe-28a5d0d18374 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:34.110378 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c187564-de83-4f76-a5fe-28a5d0d18374 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:34.121769 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1235/2482] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:39:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:39:34.125933 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988917e5-5c09-4fb4-803e-5a592e47d7ae None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:34.134978 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988917e5-5c09-4fb4-803e-5a592e47d7ae None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:34.140146 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b16337ed-9031-4d15-822c-e7b81691db13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:34.140738 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16337ed-9031-4d15-822c-e7b81691db13 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:34.141018 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16337ed-9031-4d15-822c-e7b81691db13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:34.141321 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16337ed-9031-4d15-822c-e7b81691db13 admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:34.141953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b16337ed-9031-4d15-822c-e7b81691db13 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:34.143841 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b16337ed-9031-4d15-822c-e7b81691db13 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:34.171549 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1248/2483] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:39:34 2022] POST /identity/v3/projects => generated 373 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:39:34.175683 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c187564-de83-4f76-a5fe-28a5d0d18374 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:34.187204 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c187564-de83-4f76-a5fe-28a5d0d18374 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:34.232842 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:39:34.251889 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfabd3a8-9339-4081-8e1a-d93d6d86c8e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:34.252496 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfabd3a8-9339-4081-8e1a-d93d6d86c8e5 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:34.252768 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfabd3a8-9339-4081-8e1a-d93d6d86c8e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:34.253034 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfabd3a8-9339-4081-8e1a-d93d6d86c8e5 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:34.253708 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfabd3a8-9339-4081-8e1a-d93d6d86c8e5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:34.255641 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfabd3a8-9339-4081-8e1a-d93d6d86c8e5 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:34.281980 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1236/2484] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:39:34 2022] POST /identity/v3/users => generated 287 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:39:34.283246 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b16337ed-9031-4d15-822c-e7b81691db13 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:34.293730 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b16337ed-9031-4d15-822c-e7b81691db13 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:34.299176 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b9ea55e-1665-493f-a465-f97387b96c50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:34.299782 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9ea55e-1665-493f-a465-f97387b96c50 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:34.300058 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9ea55e-1665-493f-a465-f97387b96c50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:34.300338 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9ea55e-1665-493f-a465-f97387b96c50 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:34.300952 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b9ea55e-1665-493f-a465-f97387b96c50 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:34.302623 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b9ea55e-1665-493f-a465-f97387b96c50 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:34.314228 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1249/2485] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:39:34 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 27 01:39:34.318239 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfabd3a8-9339-4081-8e1a-d93d6d86c8e5 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:34.328645 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfabd3a8-9339-4081-8e1a-d93d6d86c8e5 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:34.333171 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c8d983-ce08-4d68-8872-a1bf45e5c257 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:34.333758 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c8d983-ce08-4d68-8872-a1bf45e5c257 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:34.334036 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c8d983-ce08-4d68-8872-a1bf45e5c257 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:34.334304 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c8d983-ce08-4d68-8872-a1bf45e5c257 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:34.334933 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c8d983-ce08-4d68-8872-a1bf45e5c257 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:34.336507 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c8d983-ce08-4d68-8872-a1bf45e5c257 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:34.347070 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1237/2486] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:39:34 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:39:34.351059 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b9ea55e-1665-493f-a465-f97387b96c50 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:34.361053 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b9ea55e-1665-493f-a465-f97387b96c50 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:34.365775 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-720e69d7-5a00-438e-a04c-fa2cd14a217d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:34.366418 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720e69d7-5a00-438e-a04c-fa2cd14a217d admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:34.366697 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720e69d7-5a00-438e-a04c-fa2cd14a217d admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:34.366970 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720e69d7-5a00-438e-a04c-fa2cd14a217d admin admin] PATH_INFO: `/v3/projects/6f14e0b4a1ec4e9fa4c4f4355f5fa328/users/176e48763d6447c6ba6afcfc1705f1dc/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:34.382201 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-720e69d7-5a00-438e-a04c-fa2cd14a217d admin admin] RBAC: Authorizing `identity:create_grant(project_id=6f14e0b4a1ec4e9fa4c4f4355f5fa328, user_id=176e48763d6447c6ba6afcfc1705f1dc, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:34.384844 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-720e69d7-5a00-438e-a04c-fa2cd14a217d admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:34.398550 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1250/2487] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:39:34 2022] PUT /identity/v3/projects/6f14e0b4a1ec4e9fa4c4f4355f5fa328/users/176e48763d6447c6ba6afcfc1705f1dc/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:39:34.409044 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07686746-f4c6-4491-9d21-3fc66662cc52 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:34.409350 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07686746-f4c6-4491-9d21-3fc66662cc52 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:34.409641 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07686746-f4c6-4491-9d21-3fc66662cc52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:34.438629 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-07686746-f4c6-4491-9d21-3fc66662cc52 None None] MFA Rules not processed for user `176e48763d6447c6ba6afcfc1705f1dc`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:39:34.439843 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-07686746-f4c6-4491-9d21-3fc66662cc52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:39:34.502915 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1238/2488] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:39:34 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:39:34.514220 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-720e69d7-5a00-438e-a04c-fa2cd14a217d admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:34.516587 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-720e69d7-5a00-438e-a04c-fa2cd14a217d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:39:34.537901 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:39:34.592444 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-720e69d7-5a00-438e-a04c-fa2cd14a217d admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:34.597863 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-568b518a-e43f-40f8-a1da-c55022282a26 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:34.598454 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568b518a-e43f-40f8-a1da-c55022282a26 service nova] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:34.598729 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568b518a-e43f-40f8-a1da-c55022282a26 service nova] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:34.598997 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568b518a-e43f-40f8-a1da-c55022282a26 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:34.606515 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-568b518a-e43f-40f8-a1da-c55022282a26 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:34.608103 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-568b518a-e43f-40f8-a1da-c55022282a26 service nova] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:34.616636 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1251/2489] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:39:34 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:39:37.284080 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07686746-f4c6-4491-9d21-3fc66662cc52 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:37.344236 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07686746-f4c6-4491-9d21-3fc66662cc52 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:37.348738 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f7f3a0-fef2-4d6b-8509-076de5b318a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:37.349403 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f7f3a0-fef2-4d6b-8509-076de5b318a3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:37.349686 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f7f3a0-fef2-4d6b-8509-076de5b318a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:37.349959 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f7f3a0-fef2-4d6b-8509-076de5b318a3 admin admin] PATH_INFO: `/v3/users/176e48763d6447c6ba6afcfc1705f1dc` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:37.351095 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44f7f3a0-fef2-4d6b-8509-076de5b318a3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=176e48763d6447c6ba6afcfc1705f1dc)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:37.353015 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44f7f3a0-fef2-4d6b-8509-076de5b318a3 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:37.398997 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-44f7f3a0-fef2-4d6b-8509-076de5b318a3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '176e48763d6447c6ba6afcfc1705f1dc'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:37.405873 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-44f7f3a0-fef2-4d6b-8509-076de5b318a3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '176e48763d6447c6ba6afcfc1705f1dc'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:37.406534 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-44f7f3a0-fef2-4d6b-8509-076de5b318a3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '176e48763d6447c6ba6afcfc1705f1dc'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:37.408420 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=88,keystone:INSERT=4,keystone:DELETE=8 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:39:37.420301 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-44f7f3a0-fef2-4d6b-8509-076de5b318a3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '176e48763d6447c6ba6afcfc1705f1dc'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:37.427374 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-44f7f3a0-fef2-4d6b-8509-076de5b318a3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '176e48763d6447c6ba6afcfc1705f1dc'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:37.433471 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1239/2490] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:39:37 2022] DELETE /identity/v3/users/176e48763d6447c6ba6afcfc1705f1dc => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:39:37.438411 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32247b4e-f998-4eb3-ae9e-3097d7feba4c None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:37.438862 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32247b4e-f998-4eb3-ae9e-3097d7feba4c None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:37.438975 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32247b4e-f998-4eb3-ae9e-3097d7feba4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:37.457827 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-32247b4e-f998-4eb3-ae9e-3097d7feba4c None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:39:37.459023 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-32247b4e-f998-4eb3-ae9e-3097d7feba4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:39:37.517309 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1252/2491] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:39:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:39:37.527942 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f7f3a0-fef2-4d6b-8509-076de5b318a3 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:37.529367 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-44f7f3a0-fef2-4d6b-8509-076de5b318a3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:39:37.585883 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f7f3a0-fef2-4d6b-8509-076de5b318a3 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:37.589518 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a652418f-9645-48fd-a1a1-f362f7d65236 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:37.590115 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a652418f-9645-48fd-a1a1-f362f7d65236 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:37.590395 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a652418f-9645-48fd-a1a1-f362f7d65236 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:37.590667 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a652418f-9645-48fd-a1a1-f362f7d65236 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:37.602739 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a652418f-9645-48fd-a1a1-f362f7d65236 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:37.604212 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a652418f-9645-48fd-a1a1-f362f7d65236 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:37.613744 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1240/2492] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:39:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:39:37.885604 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32247b4e-f998-4eb3-ae9e-3097d7feba4c None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:37.887055 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-32247b4e-f998-4eb3-ae9e-3097d7feba4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:39:37.900040 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32247b4e-f998-4eb3-ae9e-3097d7feba4c None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:37.904891 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723d5316-fa77-4955-aee2-89be0ea1fd61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:37.905624 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723d5316-fa77-4955-aee2-89be0ea1fd61 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:37.905902 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723d5316-fa77-4955-aee2-89be0ea1fd61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:37.906168 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723d5316-fa77-4955-aee2-89be0ea1fd61 admin admin] PATH_INFO: `/v3/projects/6f14e0b4a1ec4e9fa4c4f4355f5fa328` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:37.907311 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723d5316-fa77-4955-aee2-89be0ea1fd61 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6f14e0b4a1ec4e9fa4c4f4355f5fa328)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:37.909248 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723d5316-fa77-4955-aee2-89be0ea1fd61 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:37.961777 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-723d5316-fa77-4955-aee2-89be0ea1fd61 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6f14e0b4a1ec4e9fa4c4f4355f5fa328', 'request_id': 'req-723d5316-fa77-4955-aee2-89be0ea1fd61'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:37.969447 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-723d5316-fa77-4955-aee2-89be0ea1fd61 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6f14e0b4a1ec4e9fa4c4f4355f5fa328', 'request_id': 'req-723d5316-fa77-4955-aee2-89be0ea1fd61'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:37.975625 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-723d5316-fa77-4955-aee2-89be0ea1fd61 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6f14e0b4a1ec4e9fa4c4f4355f5fa328', 'request_id': 'req-723d5316-fa77-4955-aee2-89be0ea1fd61'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:37.986427 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-723d5316-fa77-4955-aee2-89be0ea1fd61 admin admin] The token cache is being invalidate because project 6f14e0b4a1ec4e9fa4c4f4355f5fa328 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:39:37.986717 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-723d5316-fa77-4955-aee2-89be0ea1fd61 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:37.987359 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-723d5316-fa77-4955-aee2-89be0ea1fd61 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:37.988865 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1253/2493] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:39:37 2022] DELETE /identity/v3/projects/6f14e0b4a1ec4e9fa4c4f4355f5fa328 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:39:37.993864 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3275dcfa-8496-4aa6-9344-79c02928c437 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:37.994156 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3275dcfa-8496-4aa6-9344-79c02928c437 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:37.994418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3275dcfa-8496-4aa6-9344-79c02928c437 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:38.015478 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-3275dcfa-8496-4aa6-9344-79c02928c437 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:39:38.016697 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-3275dcfa-8496-4aa6-9344-79c02928c437 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:39:38.075248 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1241/2494] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:39:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:39:38.083158 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5a4827-3d92-4782-92bb-a8bd6856c85e None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:38.083445 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5a4827-3d92-4782-92bb-a8bd6856c85e None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:38.083723 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5a4827-3d92-4782-92bb-a8bd6856c85e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:38.103794 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-fb5a4827-3d92-4782-92bb-a8bd6856c85e None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:39:38.104996 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-fb5a4827-3d92-4782-92bb-a8bd6856c85e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:39:38.113634 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1254/2495] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:39:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:39:38.117764 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3275dcfa-8496-4aa6-9344-79c02928c437 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:38.127148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3275dcfa-8496-4aa6-9344-79c02928c437 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:38.132058 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34aa31de-b934-4bef-8fb4-c10ab4b1c741 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:38.132638 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34aa31de-b934-4bef-8fb4-c10ab4b1c741 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:38.132916 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34aa31de-b934-4bef-8fb4-c10ab4b1c741 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:38.133217 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34aa31de-b934-4bef-8fb4-c10ab4b1c741 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:38.133845 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34aa31de-b934-4bef-8fb4-c10ab4b1c741 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:39:38.134141 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34aa31de-b934-4bef-8fb4-c10ab4b1c741 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:38.135701 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34aa31de-b934-4bef-8fb4-c10ab4b1c741 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:38.147195 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1242/2496] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:39:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:39:38.151463 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5a4827-3d92-4782-92bb-a8bd6856c85e None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:38.161252 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5a4827-3d92-4782-92bb-a8bd6856c85e None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:38.165929 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6addcaf3-cb1b-46b2-ada4-b158242b3da4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:38.166515 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6addcaf3-cb1b-46b2-ada4-b158242b3da4 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:38.166792 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6addcaf3-cb1b-46b2-ada4-b158242b3da4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:38.167059 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6addcaf3-cb1b-46b2-ada4-b158242b3da4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:38.167687 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6addcaf3-cb1b-46b2-ada4-b158242b3da4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:38.169605 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6addcaf3-cb1b-46b2-ada4-b158242b3da4 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:38.191020 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:39:38.199602 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1255/2497] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:39:38 2022] POST /identity/v3/projects => generated 403 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:39:38.203708 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34aa31de-b934-4bef-8fb4-c10ab4b1c741 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:38.213621 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34aa31de-b934-4bef-8fb4-c10ab4b1c741 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:38.269415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f242dfe0-331d-491c-96af-bf493c4d360c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:38.270015 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f242dfe0-331d-491c-96af-bf493c4d360c admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:38.270290 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f242dfe0-331d-491c-96af-bf493c4d360c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:38.270561 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f242dfe0-331d-491c-96af-bf493c4d360c admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:38.271194 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f242dfe0-331d-491c-96af-bf493c4d360c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:38.273109 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f242dfe0-331d-491c-96af-bf493c4d360c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:38.282940 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=96,keystone:DELETE=2,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:39:38.295153 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1243/2498] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:39:38 2022] POST /identity/v3/users => generated 302 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:39:38.299304 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6addcaf3-cb1b-46b2-ada4-b158242b3da4 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:38.309590 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6addcaf3-cb1b-46b2-ada4-b158242b3da4 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:38.314652 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce984745-48e1-4d84-b22f-ee89d71509bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:38.315237 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce984745-48e1-4d84-b22f-ee89d71509bc admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:38.315513 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce984745-48e1-4d84-b22f-ee89d71509bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:38.315782 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce984745-48e1-4d84-b22f-ee89d71509bc admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:38.316410 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce984745-48e1-4d84-b22f-ee89d71509bc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:38.318027 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce984745-48e1-4d84-b22f-ee89d71509bc admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:38.328301 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1256/2499] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:39:38 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:39:38.331990 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f242dfe0-331d-491c-96af-bf493c4d360c admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:38.341568 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f242dfe0-331d-491c-96af-bf493c4d360c admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:38.346182 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3dc183-7655-49c2-988b-12ad1657eef3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:38.347073 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3dc183-7655-49c2-988b-12ad1657eef3 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:38.347343 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3dc183-7655-49c2-988b-12ad1657eef3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:38.347610 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3dc183-7655-49c2-988b-12ad1657eef3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:38.348264 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc3dc183-7655-49c2-988b-12ad1657eef3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:38.349871 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc3dc183-7655-49c2-988b-12ad1657eef3 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:38.360066 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1244/2500] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:39:38 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:39:38.363947 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce984745-48e1-4d84-b22f-ee89d71509bc admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:38.373596 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce984745-48e1-4d84-b22f-ee89d71509bc admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:38.378658 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d53287-c657-4972-b418-aa6b17aa22fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:38.379289 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d53287-c657-4972-b418-aa6b17aa22fa admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:38.379563 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d53287-c657-4972-b418-aa6b17aa22fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:38.379835 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d53287-c657-4972-b418-aa6b17aa22fa admin admin] PATH_INFO: `/v3/projects/a764a825bbd64b65a8635aa979667d5e/users/58ae503e197f48adae75955374d4168b/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:38.394630 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7d53287-c657-4972-b418-aa6b17aa22fa admin admin] RBAC: Authorizing `identity:create_grant(project_id=a764a825bbd64b65a8635aa979667d5e, user_id=58ae503e197f48adae75955374d4168b, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:38.397293 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7d53287-c657-4972-b418-aa6b17aa22fa admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:38.410364 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1257/2501] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:39:38 2022] PUT /identity/v3/projects/a764a825bbd64b65a8635aa979667d5e/users/58ae503e197f48adae75955374d4168b/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:39:38.415279 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfce48dd-5f25-4fd1-b699-3f4f5d38c0ad None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:38.415559 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfce48dd-5f25-4fd1-b699-3f4f5d38c0ad None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:38.415828 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfce48dd-5f25-4fd1-b699-3f4f5d38c0ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:38.434988 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-dfce48dd-5f25-4fd1-b699-3f4f5d38c0ad None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:39:38.436168 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-dfce48dd-5f25-4fd1-b699-3f4f5d38c0ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:39:38.494205 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1245/2502] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:39:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:39:38.504622 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d53287-c657-4972-b418-aa6b17aa22fa admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:38.506320 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-f7d53287-c657-4972-b418-aa6b17aa22fa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:39:38.564825 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d53287-c657-4972-b418-aa6b17aa22fa admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:38.568505 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d77e5ee-0a61-44fe-a3a4-f0813d0517d8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:38.569131 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d77e5ee-0a61-44fe-a3a4-f0813d0517d8 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:38.569413 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d77e5ee-0a61-44fe-a3a4-f0813d0517d8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:38.569680 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d77e5ee-0a61-44fe-a3a4-f0813d0517d8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:38.578849 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d77e5ee-0a61-44fe-a3a4-f0813d0517d8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:38.580211 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d77e5ee-0a61-44fe-a3a4-f0813d0517d8 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:38.589534 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1258/2503] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:39:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:39:39.207668 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7509493-1c35-486e-8800-eb6492992054 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:39.207953 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7509493-1c35-486e-8800-eb6492992054 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:39.208231 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7509493-1c35-486e-8800-eb6492992054 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:39.236611 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-a7509493-1c35-486e-8800-eb6492992054 None None] MFA Rules not processed for user `58ae503e197f48adae75955374d4168b`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:39:39.237860 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-a7509493-1c35-486e-8800-eb6492992054 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:39:39.306462 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1246/2504] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:39:39 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:39:39.310989 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d77e5ee-0a61-44fe-a3a4-f0813d0517d8 service neutron] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:39.320842 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d77e5ee-0a61-44fe-a3a4-f0813d0517d8 service neutron] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:39.325497 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e2f4ef-5127-4de9-929b-d0c4bba789b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:39.326095 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e2f4ef-5127-4de9-929b-d0c4bba789b2 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:39.326369 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e2f4ef-5127-4de9-929b-d0c4bba789b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:39.326638 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e2f4ef-5127-4de9-929b-d0c4bba789b2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:39.328060 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e2f4ef-5127-4de9-929b-d0c4bba789b2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:39.329940 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e2f4ef-5127-4de9-929b-d0c4bba789b2 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:39.361090 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1259/2505] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:39:39 2022] POST /identity/v3/projects => generated 405 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:39:39.363178 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7509493-1c35-486e-8800-eb6492992054 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:39.373580 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7509493-1c35-486e-8800-eb6492992054 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:39.429510 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-532fceb7-ab48-444d-9d5d-a3fc00abeb04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:39.430115 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532fceb7-ab48-444d-9d5d-a3fc00abeb04 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:39.430647 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532fceb7-ab48-444d-9d5d-a3fc00abeb04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:39.430915 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532fceb7-ab48-444d-9d5d-a3fc00abeb04 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:39.431552 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-532fceb7-ab48-444d-9d5d-a3fc00abeb04 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:39.433489 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-532fceb7-ab48-444d-9d5d-a3fc00abeb04 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:39.454696 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1247/2506] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:39:39 2022] POST /identity/v3/users => generated 309 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 27 01:39:39.458703 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e2f4ef-5127-4de9-929b-d0c4bba789b2 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:39.468950 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e2f4ef-5127-4de9-929b-d0c4bba789b2 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:39.473652 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e98b9ec4-007c-4be9-8cfb-a3d32596c94b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:39.474259 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98b9ec4-007c-4be9-8cfb-a3d32596c94b admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:39.474536 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98b9ec4-007c-4be9-8cfb-a3d32596c94b admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:39.474801 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98b9ec4-007c-4be9-8cfb-a3d32596c94b admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:39.475433 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e98b9ec4-007c-4be9-8cfb-a3d32596c94b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:39.477019 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e98b9ec4-007c-4be9-8cfb-a3d32596c94b admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:39.488066 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1260/2507] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:39:39 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:39:39.491995 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-532fceb7-ab48-444d-9d5d-a3fc00abeb04 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:39.497099 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:39:39.504193 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-532fceb7-ab48-444d-9d5d-a3fc00abeb04 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:39.509332 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99857d07-161c-45ec-9da5-503c19f421b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:39.509980 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99857d07-161c-45ec-9da5-503c19f421b8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:39.510262 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99857d07-161c-45ec-9da5-503c19f421b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:39.510589 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99857d07-161c-45ec-9da5-503c19f421b8 admin admin] PATH_INFO: `/v3/projects/17465ed036c84d0ab3ec9387dfc52257/users/f5dae6d943684c24a1fc65795d6a245c/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:39.526707 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99857d07-161c-45ec-9da5-503c19f421b8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=17465ed036c84d0ab3ec9387dfc52257, user_id=f5dae6d943684c24a1fc65795d6a245c, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:39.529418 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99857d07-161c-45ec-9da5-503c19f421b8 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:39.542642 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1248/2508] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:39:39 2022] PUT /identity/v3/projects/17465ed036c84d0ab3ec9387dfc52257/users/f5dae6d943684c24a1fc65795d6a245c/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:39:39.546894 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e98b9ec4-007c-4be9-8cfb-a3d32596c94b admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:39.557291 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e98b9ec4-007c-4be9-8cfb-a3d32596c94b admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:39.614898 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9310f29d-e406-49ed-b76f-7d051505615d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:39.615528 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9310f29d-e406-49ed-b76f-7d051505615d admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:39.615805 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9310f29d-e406-49ed-b76f-7d051505615d admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:39.616075 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9310f29d-e406-49ed-b76f-7d051505615d admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:39.616707 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9310f29d-e406-49ed-b76f-7d051505615d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:39.618334 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9310f29d-e406-49ed-b76f-7d051505615d admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:39.628370 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1261/2509] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:39:39 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:39:39.632618 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99857d07-161c-45ec-9da5-503c19f421b8 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:39.643469 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99857d07-161c-45ec-9da5-503c19f421b8 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:39.648074 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0929dee-3fd3-4dce-ace6-6cb492748178 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:39.648710 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0929dee-3fd3-4dce-ace6-6cb492748178 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:39.648988 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0929dee-3fd3-4dce-ace6-6cb492748178 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:39.649298 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0929dee-3fd3-4dce-ace6-6cb492748178 admin admin] PATH_INFO: `/v3/projects/17465ed036c84d0ab3ec9387dfc52257/users/f5dae6d943684c24a1fc65795d6a245c/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:39.651224 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0929dee-3fd3-4dce-ace6-6cb492748178 admin admin] RBAC: Authorizing `identity:create_grant(project_id=17465ed036c84d0ab3ec9387dfc52257, user_id=f5dae6d943684c24a1fc65795d6a245c, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:39.653905 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0929dee-3fd3-4dce-ace6-6cb492748178 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:39.663903 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1249/2510] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:39:39 2022] PUT /identity/v3/projects/17465ed036c84d0ab3ec9387dfc52257/users/f5dae6d943684c24a1fc65795d6a245c/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:39:40.308770 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa6dd67-44a2-415d-b7c1-61c2945d5784 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:40.309076 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa6dd67-44a2-415d-b7c1-61c2945d5784 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:40.309358 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa6dd67-44a2-415d-b7c1-61c2945d5784 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:40.337556 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-8aa6dd67-44a2-415d-b7c1-61c2945d5784 None None] MFA Rules not processed for user `f5dae6d943684c24a1fc65795d6a245c`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:39:40.338798 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-8aa6dd67-44a2-415d-b7c1-61c2945d5784 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:39:40.357843 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:39:40.414544 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1262/2511] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:39:40 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:39:41.703184 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0929dee-3fd3-4dce-ace6-6cb492748178 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:41.764375 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0929dee-3fd3-4dce-ace6-6cb492748178 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:41.769580 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d888bc4d-69c3-4c25-be81-293a802302cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:41.770192 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d888bc4d-69c3-4c25-be81-293a802302cf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:41.770466 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d888bc4d-69c3-4c25-be81-293a802302cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:41.770732 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d888bc4d-69c3-4c25-be81-293a802302cf admin admin] PATH_INFO: `/v3/users/58ae503e197f48adae75955374d4168b` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:41.771855 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d888bc4d-69c3-4c25-be81-293a802302cf admin admin] RBAC: Authorizing `identity:delete_user(user_id=58ae503e197f48adae75955374d4168b)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:41.773785 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d888bc4d-69c3-4c25-be81-293a802302cf admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:41.820714 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d888bc4d-69c3-4c25-be81-293a802302cf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '58ae503e197f48adae75955374d4168b'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:41.827004 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d888bc4d-69c3-4c25-be81-293a802302cf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '58ae503e197f48adae75955374d4168b'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:41.828278 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d888bc4d-69c3-4c25-be81-293a802302cf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '58ae503e197f48adae75955374d4168b'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:41.840127 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d888bc4d-69c3-4c25-be81-293a802302cf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '58ae503e197f48adae75955374d4168b'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:41.847218 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d888bc4d-69c3-4c25-be81-293a802302cf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '58ae503e197f48adae75955374d4168b'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:41.852921 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1250/2512] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:39:41 2022] DELETE /identity/v3/users/58ae503e197f48adae75955374d4168b => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:39:42.031062 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa6dd67-44a2-415d-b7c1-61c2945d5784 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:42.042370 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa6dd67-44a2-415d-b7c1-61c2945d5784 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:42.102683 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9dd75c-d5a6-46ed-b801-cb637d9293d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:42.103261 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9dd75c-d5a6-46ed-b801-cb637d9293d2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:42.103537 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9dd75c-d5a6-46ed-b801-cb637d9293d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:42.103802 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9dd75c-d5a6-46ed-b801-cb637d9293d2 admin admin] PATH_INFO: `/v3/projects/a764a825bbd64b65a8635aa979667d5e` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:42.104919 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac9dd75c-d5a6-46ed-b801-cb637d9293d2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a764a825bbd64b65a8635aa979667d5e)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:42.106839 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac9dd75c-d5a6-46ed-b801-cb637d9293d2 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:42.153821 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-ac9dd75c-d5a6-46ed-b801-cb637d9293d2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a764a825bbd64b65a8635aa979667d5e', 'request_id': 'req-ac9dd75c-d5a6-46ed-b801-cb637d9293d2'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:42.161227 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-ac9dd75c-d5a6-46ed-b801-cb637d9293d2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a764a825bbd64b65a8635aa979667d5e', 'request_id': 'req-ac9dd75c-d5a6-46ed-b801-cb637d9293d2'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:42.167381 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-ac9dd75c-d5a6-46ed-b801-cb637d9293d2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a764a825bbd64b65a8635aa979667d5e', 'request_id': 'req-ac9dd75c-d5a6-46ed-b801-cb637d9293d2'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:42.178582 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-ac9dd75c-d5a6-46ed-b801-cb637d9293d2 admin admin] The token cache is being invalidate because project a764a825bbd64b65a8635aa979667d5e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:39:42.178887 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-ac9dd75c-d5a6-46ed-b801-cb637d9293d2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:42.180151 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-ac9dd75c-d5a6-46ed-b801-cb637d9293d2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:42.181906 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1263/2513] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:39:42 2022] DELETE /identity/v3/projects/a764a825bbd64b65a8635aa979667d5e => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:39:42.185806 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d888bc4d-69c3-4c25-be81-293a802302cf admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:42.187732 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d888bc4d-69c3-4c25-be81-293a802302cf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:39:42.200599 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d888bc4d-69c3-4c25-be81-293a802302cf admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:42.241822 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=87,keystone:INSERT=3,keystone:DELETE=10 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:39:42.258030 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57bccbc-d6c4-432f-bc9e-241a824a220a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:42.258679 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57bccbc-d6c4-432f-bc9e-241a824a220a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:42.258956 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57bccbc-d6c4-432f-bc9e-241a824a220a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:42.259225 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57bccbc-d6c4-432f-bc9e-241a824a220a admin admin] PATH_INFO: `/v3/users/f5dae6d943684c24a1fc65795d6a245c` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:42.260377 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f57bccbc-d6c4-432f-bc9e-241a824a220a admin admin] RBAC: Authorizing `identity:delete_user(user_id=f5dae6d943684c24a1fc65795d6a245c)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:42.262323 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f57bccbc-d6c4-432f-bc9e-241a824a220a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:42.309236 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f57bccbc-d6c4-432f-bc9e-241a824a220a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f5dae6d943684c24a1fc65795d6a245c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:42.318003 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f57bccbc-d6c4-432f-bc9e-241a824a220a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f5dae6d943684c24a1fc65795d6a245c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:42.318655 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f57bccbc-d6c4-432f-bc9e-241a824a220a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f5dae6d943684c24a1fc65795d6a245c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:42.330892 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f57bccbc-d6c4-432f-bc9e-241a824a220a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f5dae6d943684c24a1fc65795d6a245c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:42.337764 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f57bccbc-d6c4-432f-bc9e-241a824a220a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f5dae6d943684c24a1fc65795d6a245c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:42.343521 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1251/2514] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:39:42 2022] DELETE /identity/v3/users/f5dae6d943684c24a1fc65795d6a245c => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:39:42.514585 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9dd75c-d5a6-46ed-b801-cb637d9293d2 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:42.516699 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-ac9dd75c-d5a6-46ed-b801-cb637d9293d2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:39:42.530692 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9dd75c-d5a6-46ed-b801-cb637d9293d2 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:42.574843 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:39:42.591861 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b3cb70f-93c9-4ca9-94a4-ffbb44106565 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:42.592613 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3cb70f-93c9-4ca9-94a4-ffbb44106565 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:42.593471 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3cb70f-93c9-4ca9-94a4-ffbb44106565 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:42.593733 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3cb70f-93c9-4ca9-94a4-ffbb44106565 admin admin] PATH_INFO: `/v3/projects/17465ed036c84d0ab3ec9387dfc52257` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:42.594859 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b3cb70f-93c9-4ca9-94a4-ffbb44106565 admin admin] RBAC: Authorizing `identity:delete_project(project_id=17465ed036c84d0ab3ec9387dfc52257)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:42.596920 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b3cb70f-93c9-4ca9-94a4-ffbb44106565 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:42.644903 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-0b3cb70f-93c9-4ca9-94a4-ffbb44106565 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '17465ed036c84d0ab3ec9387dfc52257', 'request_id': 'req-0b3cb70f-93c9-4ca9-94a4-ffbb44106565'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:42.651528 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-0b3cb70f-93c9-4ca9-94a4-ffbb44106565 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '17465ed036c84d0ab3ec9387dfc52257', 'request_id': 'req-0b3cb70f-93c9-4ca9-94a4-ffbb44106565'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:42.658200 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-0b3cb70f-93c9-4ca9-94a4-ffbb44106565 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '17465ed036c84d0ab3ec9387dfc52257', 'request_id': 'req-0b3cb70f-93c9-4ca9-94a4-ffbb44106565'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:42.668360 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-0b3cb70f-93c9-4ca9-94a4-ffbb44106565 admin admin] The token cache is being invalidate because project 17465ed036c84d0ab3ec9387dfc52257 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:39:42.668642 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-0b3cb70f-93c9-4ca9-94a4-ffbb44106565 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:42.669295 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-0b3cb70f-93c9-4ca9-94a4-ffbb44106565 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:39:42.670760 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1264/2515] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:39:42 2022] DELETE /identity/v3/projects/17465ed036c84d0ab3ec9387dfc52257 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:39:42.676253 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a013005f-8b17-40b6-8432-ca3f7e6e8e42 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:42.676538 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a013005f-8b17-40b6-8432-ca3f7e6e8e42 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:42.676807 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a013005f-8b17-40b6-8432-ca3f7e6e8e42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:42.696436 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-a013005f-8b17-40b6-8432-ca3f7e6e8e42 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:39:42.697667 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-a013005f-8b17-40b6-8432-ca3f7e6e8e42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:39:42.759770 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1252/2516] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:39:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:39:42.767500 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d758a384-4c11-45de-92bc-ea7410807885 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:42.767776 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d758a384-4c11-45de-92bc-ea7410807885 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:42.768042 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d758a384-4c11-45de-92bc-ea7410807885 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:42.787711 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-d758a384-4c11-45de-92bc-ea7410807885 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:39:42.788903 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-d758a384-4c11-45de-92bc-ea7410807885 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:39:42.797623 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1265/2517] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:39:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:39:42.801761 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a013005f-8b17-40b6-8432-ca3f7e6e8e42 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:42.812084 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a013005f-8b17-40b6-8432-ca3f7e6e8e42 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:42.817437 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b89d207-0d8e-45a8-b7d0-a0435657b445 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:42.818081 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b89d207-0d8e-45a8-b7d0-a0435657b445 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:42.818394 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b89d207-0d8e-45a8-b7d0-a0435657b445 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:42.818814 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b89d207-0d8e-45a8-b7d0-a0435657b445 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:42.819551 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b89d207-0d8e-45a8-b7d0-a0435657b445 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:39:42.819969 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b89d207-0d8e-45a8-b7d0-a0435657b445 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:42.821809 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b89d207-0d8e-45a8-b7d0-a0435657b445 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:42.833816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1253/2518] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:39:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:39:42.838051 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d758a384-4c11-45de-92bc-ea7410807885 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:42.848589 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d758a384-4c11-45de-92bc-ea7410807885 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:42.853236 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be7b953-2686-4886-928c-4d81e9060836 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:42.853820 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be7b953-2686-4886-928c-4d81e9060836 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:42.854091 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be7b953-2686-4886-928c-4d81e9060836 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:42.854354 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be7b953-2686-4886-928c-4d81e9060836 admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:42.854966 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6be7b953-2686-4886-928c-4d81e9060836 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:42.856837 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6be7b953-2686-4886-928c-4d81e9060836 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:42.884306 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1266/2519] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:39:42 2022] POST /identity/v3/projects => generated 371 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:39:42.888339 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b89d207-0d8e-45a8-b7d0-a0435657b445 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:42.898731 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b89d207-0d8e-45a8-b7d0-a0435657b445 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:42.945419 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:39:42.956917 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4a991a-7183-447f-b2be-27688ec67d93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:42.957578 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4a991a-7183-447f-b2be-27688ec67d93 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:42.957853 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4a991a-7183-447f-b2be-27688ec67d93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:42.958125 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4a991a-7183-447f-b2be-27688ec67d93 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:42.958793 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4a991a-7183-447f-b2be-27688ec67d93 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:42.960696 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4a991a-7183-447f-b2be-27688ec67d93 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:42.981761 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1254/2520] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:39:42 2022] POST /identity/v3/users => generated 286 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:39:42.985861 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be7b953-2686-4886-928c-4d81e9060836 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:42.996225 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be7b953-2686-4886-928c-4d81e9060836 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:43.000843 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c4c1eb-1d91-4f88-9ff3-de762a27c5a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:43.001463 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c4c1eb-1d91-4f88-9ff3-de762a27c5a8 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:43.001743 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c4c1eb-1d91-4f88-9ff3-de762a27c5a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:43.002017 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c4c1eb-1d91-4f88-9ff3-de762a27c5a8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:43.002651 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8c4c1eb-1d91-4f88-9ff3-de762a27c5a8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:43.004262 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8c4c1eb-1d91-4f88-9ff3-de762a27c5a8 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:43.014051 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1267/2521] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:39:42 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:39:43.018673 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4a991a-7183-447f-b2be-27688ec67d93 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:43.028769 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4a991a-7183-447f-b2be-27688ec67d93 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:43.034795 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d690522-1925-44b4-a121-1ceb05d4a741 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:43.035735 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d690522-1925-44b4-a121-1ceb05d4a741 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:43.036014 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d690522-1925-44b4-a121-1ceb05d4a741 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:43.036281 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d690522-1925-44b4-a121-1ceb05d4a741 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:43.036911 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d690522-1925-44b4-a121-1ceb05d4a741 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:43.038536 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d690522-1925-44b4-a121-1ceb05d4a741 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:43.048976 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1255/2522] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:39:43 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:39:43.052965 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c4c1eb-1d91-4f88-9ff3-de762a27c5a8 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:43.063137 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c4c1eb-1d91-4f88-9ff3-de762a27c5a8 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:43.067692 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ccaa28-d9b0-4cc7-bffa-346d9e0414cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:43.068329 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ccaa28-d9b0-4cc7-bffa-346d9e0414cc admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:43.068606 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ccaa28-d9b0-4cc7-bffa-346d9e0414cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:43.068871 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ccaa28-d9b0-4cc7-bffa-346d9e0414cc admin admin] PATH_INFO: `/v3/projects/5b4352570da64fa2976ca889536e4ef3/users/01b747ce31a8435abebba9fa7b1f8c7a/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:43.083677 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ccaa28-d9b0-4cc7-bffa-346d9e0414cc admin admin] RBAC: Authorizing `identity:create_grant(project_id=5b4352570da64fa2976ca889536e4ef3, user_id=01b747ce31a8435abebba9fa7b1f8c7a, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:43.086366 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ccaa28-d9b0-4cc7-bffa-346d9e0414cc admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:43.099556 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1268/2523] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:39:43 2022] PUT /identity/v3/projects/5b4352570da64fa2976ca889536e4ef3/users/01b747ce31a8435abebba9fa7b1f8c7a/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:39:43.162748 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf9c7a4-482d-4cd2-9771-3440506f079a None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:43.162963 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf9c7a4-482d-4cd2-9771-3440506f079a None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:43.163200 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf9c7a4-482d-4cd2-9771-3440506f079a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:43.190765 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-7cf9c7a4-482d-4cd2-9771-3440506f079a None None] MFA Rules not processed for user `01b747ce31a8435abebba9fa7b1f8c7a`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:39:43.191964 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-7cf9c7a4-482d-4cd2-9771-3440506f079a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:39:43.255170 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1256/2524] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:39:43 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:39:43.259327 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ccaa28-d9b0-4cc7-bffa-346d9e0414cc admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:43.319749 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ccaa28-d9b0-4cc7-bffa-346d9e0414cc admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:43.324516 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b31fcea8-c60d-404f-9e5f-6d72005bfd0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:43.325139 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31fcea8-c60d-404f-9e5f-6d72005bfd0f admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:43.325423 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31fcea8-c60d-404f-9e5f-6d72005bfd0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:43.325696 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31fcea8-c60d-404f-9e5f-6d72005bfd0f admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:43.326318 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b31fcea8-c60d-404f-9e5f-6d72005bfd0f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:43.328206 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b31fcea8-c60d-404f-9e5f-6d72005bfd0f admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:43.343577 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:39:43.356830 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1269/2525] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:39:43 2022] POST /identity/v3/projects => generated 371 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:39:43.361155 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf9c7a4-482d-4cd2-9771-3440506f079a None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:43.371626 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf9c7a4-482d-4cd2-9771-3440506f079a None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:43.426751 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a16b20-ba4b-4d52-99bf-6a8e93fb54ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:43.427358 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a16b20-ba4b-4d52-99bf-6a8e93fb54ff admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:43.427639 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a16b20-ba4b-4d52-99bf-6a8e93fb54ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:43.427911 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a16b20-ba4b-4d52-99bf-6a8e93fb54ff admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:43.428536 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a16b20-ba4b-4d52-99bf-6a8e93fb54ff admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:43.430731 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a16b20-ba4b-4d52-99bf-6a8e93fb54ff admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:43.451752 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1257/2526] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:39:43 2022] POST /identity/v3/users => generated 292 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:39:43.456042 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b31fcea8-c60d-404f-9e5f-6d72005bfd0f admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:43.466273 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b31fcea8-c60d-404f-9e5f-6d72005bfd0f admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:43.471395 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521b05ab-2e05-48be-8955-a7dab71d590d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:43.471985 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521b05ab-2e05-48be-8955-a7dab71d590d admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:43.472266 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521b05ab-2e05-48be-8955-a7dab71d590d admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:43.472538 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521b05ab-2e05-48be-8955-a7dab71d590d admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:43.473212 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-521b05ab-2e05-48be-8955-a7dab71d590d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:43.474796 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-521b05ab-2e05-48be-8955-a7dab71d590d admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:43.485749 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1270/2527] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:39:43 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:39:43.489784 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a16b20-ba4b-4d52-99bf-6a8e93fb54ff admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:43.499623 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a16b20-ba4b-4d52-99bf-6a8e93fb54ff admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:43.504806 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1db7274-2f4c-4ce5-bb94-0253ea7f0560 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:43.505472 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1db7274-2f4c-4ce5-bb94-0253ea7f0560 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:43.505752 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1db7274-2f4c-4ce5-bb94-0253ea7f0560 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:43.506017 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1db7274-2f4c-4ce5-bb94-0253ea7f0560 admin admin] PATH_INFO: `/v3/projects/6fa76a593a5e451ab2f3f0de2d6a940c/users/ba96ebe0cbe44961b4091f27de8d3be5/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:43.521599 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1db7274-2f4c-4ce5-bb94-0253ea7f0560 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6fa76a593a5e451ab2f3f0de2d6a940c, user_id=ba96ebe0cbe44961b4091f27de8d3be5, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:43.524744 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1db7274-2f4c-4ce5-bb94-0253ea7f0560 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:43.537878 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1258/2528] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:39:43 2022] PUT /identity/v3/projects/6fa76a593a5e451ab2f3f0de2d6a940c/users/ba96ebe0cbe44961b4091f27de8d3be5/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:39:43.541973 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521b05ab-2e05-48be-8955-a7dab71d590d admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:43.552582 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521b05ab-2e05-48be-8955-a7dab71d590d admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:43.608809 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04473209-7e79-4907-95fc-28d46d79fed2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:43.609459 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04473209-7e79-4907-95fc-28d46d79fed2 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:43.609798 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04473209-7e79-4907-95fc-28d46d79fed2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:43.610419 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04473209-7e79-4907-95fc-28d46d79fed2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:43.610701 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04473209-7e79-4907-95fc-28d46d79fed2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:43.612288 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04473209-7e79-4907-95fc-28d46d79fed2 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:43.622099 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1271/2529] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:39:43 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:39:43.627066 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1db7274-2f4c-4ce5-bb94-0253ea7f0560 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:43.637253 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1db7274-2f4c-4ce5-bb94-0253ea7f0560 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:43.642034 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09604939-b23b-45b4-aca7-90f53522e9aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:43.642680 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09604939-b23b-45b4-aca7-90f53522e9aa admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:43.642959 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09604939-b23b-45b4-aca7-90f53522e9aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:43.643229 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09604939-b23b-45b4-aca7-90f53522e9aa admin admin] PATH_INFO: `/v3/projects/6fa76a593a5e451ab2f3f0de2d6a940c/users/ba96ebe0cbe44961b4091f27de8d3be5/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:43.645184 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09604939-b23b-45b4-aca7-90f53522e9aa admin admin] RBAC: Authorizing `identity:create_grant(project_id=6fa76a593a5e451ab2f3f0de2d6a940c, user_id=ba96ebe0cbe44961b4091f27de8d3be5, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:43.647830 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09604939-b23b-45b4-aca7-90f53522e9aa admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:43.657965 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1259/2530] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:39:43 2022] PUT /identity/v3/projects/6fa76a593a5e451ab2f3f0de2d6a940c/users/ba96ebe0cbe44961b4091f27de8d3be5/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:39:43.667834 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ee8f57-120d-4b7b-91f5-c29043e4968a None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:43.668118 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ee8f57-120d-4b7b-91f5-c29043e4968a None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:43.668392 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ee8f57-120d-4b7b-91f5-c29043e4968a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:43.697186 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-82ee8f57-120d-4b7b-91f5-c29043e4968a None None] MFA Rules not processed for user `ba96ebe0cbe44961b4091f27de8d3be5`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:39:43.698324 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-82ee8f57-120d-4b7b-91f5-c29043e4968a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:39:43.766460 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1272/2531] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:39:43 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:39:43.777317 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09604939-b23b-45b4-aca7-90f53522e9aa admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:43.779027 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-09604939-b23b-45b4-aca7-90f53522e9aa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:39:43.793326 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:39:43.840903 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09604939-b23b-45b4-aca7-90f53522e9aa admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:43.844711 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffea6f04-07ca-4a3e-b17a-b4a7fa1da763 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:43.845335 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffea6f04-07ca-4a3e-b17a-b4a7fa1da763 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:43.845610 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffea6f04-07ca-4a3e-b17a-b4a7fa1da763 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:43.845879 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffea6f04-07ca-4a3e-b17a-b4a7fa1da763 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:43.854645 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffea6f04-07ca-4a3e-b17a-b4a7fa1da763 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:43.856098 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffea6f04-07ca-4a3e-b17a-b4a7fa1da763 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:43.866278 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1260/2532] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:39:43 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:39:43.941863 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82ee8f57-120d-4b7b-91f5-c29043e4968a None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:39:43.951532 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82ee8f57-120d-4b7b-91f5-c29043e4968a None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:39:43.955787 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cdb3cd3-e30f-46d7-8c95-b350a2f48735 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:39:43.956377 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdb3cd3-e30f-46d7-8c95-b350a2f48735 service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:39:43.956658 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdb3cd3-e30f-46d7-8c95-b350a2f48735 service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:39:43.959010 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdb3cd3-e30f-46d7-8c95-b350a2f48735 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:39:44.018172 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cdb3cd3-e30f-46d7-8c95-b350a2f48735 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:39:44.019613 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cdb3cd3-e30f-46d7-8c95-b350a2f48735 service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:39:44.028416 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1273/2533] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:39:43 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:39:53.861953 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=21 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:39:54.024385 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=98 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:40:26.490677 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffea6f04-07ca-4a3e-b17a-b4a7fa1da763 service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:26.552449 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffea6f04-07ca-4a3e-b17a-b4a7fa1da763 service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:26.558183 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a705e6a-5484-4867-8e15-b9d4d6e0b983 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:26.558791 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a705e6a-5484-4867-8e15-b9d4d6e0b983 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:26.559065 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a705e6a-5484-4867-8e15-b9d4d6e0b983 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:26.559333 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a705e6a-5484-4867-8e15-b9d4d6e0b983 admin admin] PATH_INFO: `/v3/users/01b747ce31a8435abebba9fa7b1f8c7a` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:26.560429 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a705e6a-5484-4867-8e15-b9d4d6e0b983 admin admin] RBAC: Authorizing `identity:delete_user(user_id=01b747ce31a8435abebba9fa7b1f8c7a)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:26.562361 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a705e6a-5484-4867-8e15-b9d4d6e0b983 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:26.606766 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1a705e6a-5484-4867-8e15-b9d4d6e0b983 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '01b747ce31a8435abebba9fa7b1f8c7a'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:26.613645 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1a705e6a-5484-4867-8e15-b9d4d6e0b983 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '01b747ce31a8435abebba9fa7b1f8c7a'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:26.614263 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1a705e6a-5484-4867-8e15-b9d4d6e0b983 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '01b747ce31a8435abebba9fa7b1f8c7a'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:26.626527 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1a705e6a-5484-4867-8e15-b9d4d6e0b983 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '01b747ce31a8435abebba9fa7b1f8c7a'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:26.633169 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1a705e6a-5484-4867-8e15-b9d4d6e0b983 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '01b747ce31a8435abebba9fa7b1f8c7a'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:26.639271 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1261/2534] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:40:26 2022] DELETE /identity/v3/users/01b747ce31a8435abebba9fa7b1f8c7a => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:40:26.643735 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6412b6ca-e363-4946-aa1d-0493d105f48b None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:26.644209 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6412b6ca-e363-4946-aa1d-0493d105f48b None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:26.644317 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6412b6ca-e363-4946-aa1d-0493d105f48b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:26.664385 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-6412b6ca-e363-4946-aa1d-0493d105f48b None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:40:26.665626 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-6412b6ca-e363-4946-aa1d-0493d105f48b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:40:26.723433 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1274/2535] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:40:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:40:26.734051 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a705e6a-5484-4867-8e15-b9d4d6e0b983 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:26.735436 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-1a705e6a-5484-4867-8e15-b9d4d6e0b983 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:40:26.791580 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a705e6a-5484-4867-8e15-b9d4d6e0b983 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:26.795262 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cef7fba-53db-4452-a1ff-4249b1ddede1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:26.795859 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cef7fba-53db-4452-a1ff-4249b1ddede1 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:26.796137 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cef7fba-53db-4452-a1ff-4249b1ddede1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:26.796408 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cef7fba-53db-4452-a1ff-4249b1ddede1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:26.805783 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cef7fba-53db-4452-a1ff-4249b1ddede1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:26.807636 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cef7fba-53db-4452-a1ff-4249b1ddede1 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:26.817181 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1262/2536] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:40:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:40:27.081463 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6412b6ca-e363-4946-aa1d-0493d105f48b None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:27.082913 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-6412b6ca-e363-4946-aa1d-0493d105f48b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:40:27.095866 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6412b6ca-e363-4946-aa1d-0493d105f48b None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:27.100460 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e9c1c5-3bd1-4f2b-901c-94d0da3c0ead admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:27.101126 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e9c1c5-3bd1-4f2b-901c-94d0da3c0ead admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:27.101410 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e9c1c5-3bd1-4f2b-901c-94d0da3c0ead admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:27.101680 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e9c1c5-3bd1-4f2b-901c-94d0da3c0ead admin admin] PATH_INFO: `/v3/projects/5b4352570da64fa2976ca889536e4ef3` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:27.102797 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33e9c1c5-3bd1-4f2b-901c-94d0da3c0ead admin admin] RBAC: Authorizing `identity:delete_project(project_id=5b4352570da64fa2976ca889536e4ef3)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:27.104966 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33e9c1c5-3bd1-4f2b-901c-94d0da3c0ead admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:27.153825 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-33e9c1c5-3bd1-4f2b-901c-94d0da3c0ead admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5b4352570da64fa2976ca889536e4ef3', 'request_id': 'req-33e9c1c5-3bd1-4f2b-901c-94d0da3c0ead'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:27.161581 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-33e9c1c5-3bd1-4f2b-901c-94d0da3c0ead admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5b4352570da64fa2976ca889536e4ef3', 'request_id': 'req-33e9c1c5-3bd1-4f2b-901c-94d0da3c0ead'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:27.168168 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-33e9c1c5-3bd1-4f2b-901c-94d0da3c0ead admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5b4352570da64fa2976ca889536e4ef3', 'request_id': 'req-33e9c1c5-3bd1-4f2b-901c-94d0da3c0ead'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:27.178503 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-33e9c1c5-3bd1-4f2b-901c-94d0da3c0ead admin admin] The token cache is being invalidate because project 5b4352570da64fa2976ca889536e4ef3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:40:27.178796 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-33e9c1c5-3bd1-4f2b-901c-94d0da3c0ead admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:27.179978 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-33e9c1c5-3bd1-4f2b-901c-94d0da3c0ead admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:27.182111 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1275/2537] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:40:27 2022] DELETE /identity/v3/projects/5b4352570da64fa2976ca889536e4ef3 => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:40:27.186067 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cef7fba-53db-4452-a1ff-4249b1ddede1 service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:27.195492 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cef7fba-53db-4452-a1ff-4249b1ddede1 service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:27.214389 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:40:27.253605 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4061ab50-30b4-4282-8854-fda784f69a4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:27.254216 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4061ab50-30b4-4282-8854-fda784f69a4d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:27.254490 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4061ab50-30b4-4282-8854-fda784f69a4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:27.254757 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4061ab50-30b4-4282-8854-fda784f69a4d admin admin] PATH_INFO: `/v3/users/ba96ebe0cbe44961b4091f27de8d3be5` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:27.255859 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4061ab50-30b4-4282-8854-fda784f69a4d admin admin] RBAC: Authorizing `identity:delete_user(user_id=ba96ebe0cbe44961b4091f27de8d3be5)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:27.257802 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4061ab50-30b4-4282-8854-fda784f69a4d admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:27.302491 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4061ab50-30b4-4282-8854-fda784f69a4d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ba96ebe0cbe44961b4091f27de8d3be5'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:27.310608 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4061ab50-30b4-4282-8854-fda784f69a4d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ba96ebe0cbe44961b4091f27de8d3be5'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:27.311389 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4061ab50-30b4-4282-8854-fda784f69a4d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ba96ebe0cbe44961b4091f27de8d3be5'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:27.324325 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4061ab50-30b4-4282-8854-fda784f69a4d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ba96ebe0cbe44961b4091f27de8d3be5'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:27.331156 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4061ab50-30b4-4282-8854-fda784f69a4d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ba96ebe0cbe44961b4091f27de8d3be5'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:27.336895 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1263/2538] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:40:27 2022] DELETE /identity/v3/users/ba96ebe0cbe44961b4091f27de8d3be5 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:40:27.611010 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e9c1c5-3bd1-4f2b-901c-94d0da3c0ead admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:27.612543 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-33e9c1c5-3bd1-4f2b-901c-94d0da3c0ead admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:40:27.626898 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e9c1c5-3bd1-4f2b-901c-94d0da3c0ead admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:27.684897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-476aa382-3828-4e91-80f0-7ed11903bbc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:27.685738 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476aa382-3828-4e91-80f0-7ed11903bbc2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:27.685867 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476aa382-3828-4e91-80f0-7ed11903bbc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:27.686074 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476aa382-3828-4e91-80f0-7ed11903bbc2 admin admin] PATH_INFO: `/v3/projects/6fa76a593a5e451ab2f3f0de2d6a940c` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:27.687172 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-476aa382-3828-4e91-80f0-7ed11903bbc2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6fa76a593a5e451ab2f3f0de2d6a940c)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:27.689091 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-476aa382-3828-4e91-80f0-7ed11903bbc2 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:27.719232 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:40:27.737997 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-476aa382-3828-4e91-80f0-7ed11903bbc2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6fa76a593a5e451ab2f3f0de2d6a940c', 'request_id': 'req-476aa382-3828-4e91-80f0-7ed11903bbc2'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:27.745944 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-476aa382-3828-4e91-80f0-7ed11903bbc2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6fa76a593a5e451ab2f3f0de2d6a940c', 'request_id': 'req-476aa382-3828-4e91-80f0-7ed11903bbc2'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:27.752150 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-476aa382-3828-4e91-80f0-7ed11903bbc2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6fa76a593a5e451ab2f3f0de2d6a940c', 'request_id': 'req-476aa382-3828-4e91-80f0-7ed11903bbc2'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:27.763303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-476aa382-3828-4e91-80f0-7ed11903bbc2 admin admin] The token cache is being invalidate because project 6fa76a593a5e451ab2f3f0de2d6a940c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:40:27.763588 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-476aa382-3828-4e91-80f0-7ed11903bbc2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:27.764213 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-476aa382-3828-4e91-80f0-7ed11903bbc2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:27.765710 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1276/2539] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:40:27 2022] DELETE /identity/v3/projects/6fa76a593a5e451ab2f3f0de2d6a940c => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:40:27.770590 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da98c55b-ce2a-4b95-a4fe-0efc72519293 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:27.771089 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da98c55b-ce2a-4b95-a4fe-0efc72519293 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:27.771209 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da98c55b-ce2a-4b95-a4fe-0efc72519293 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:27.790372 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-da98c55b-ce2a-4b95-a4fe-0efc72519293 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:40:27.792148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-da98c55b-ce2a-4b95-a4fe-0efc72519293 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:40:27.850172 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1264/2540] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:40:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:40:27.859021 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8cd3b5a-bc51-4487-b354-02f73d1a738f None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:27.859307 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8cd3b5a-bc51-4487-b354-02f73d1a738f None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:27.859581 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8cd3b5a-bc51-4487-b354-02f73d1a738f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:27.879666 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-e8cd3b5a-bc51-4487-b354-02f73d1a738f None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:40:27.880867 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-e8cd3b5a-bc51-4487-b354-02f73d1a738f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:40:27.888704 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1277/2541] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:40:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:40:27.892908 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da98c55b-ce2a-4b95-a4fe-0efc72519293 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:27.902773 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da98c55b-ce2a-4b95-a4fe-0efc72519293 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:27.907324 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a4ed7e-f9d2-42d7-88b7-b69af81696eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:27.907911 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a4ed7e-f9d2-42d7-88b7-b69af81696eb admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:27.908187 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a4ed7e-f9d2-42d7-88b7-b69af81696eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:27.908457 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a4ed7e-f9d2-42d7-88b7-b69af81696eb admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:27.909128 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4a4ed7e-f9d2-42d7-88b7-b69af81696eb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:40:27.909435 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4a4ed7e-f9d2-42d7-88b7-b69af81696eb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:27.911024 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4a4ed7e-f9d2-42d7-88b7-b69af81696eb admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:27.923799 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1265/2542] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:40:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:40:27.928083 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8cd3b5a-bc51-4487-b354-02f73d1a738f None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:27.938387 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8cd3b5a-bc51-4487-b354-02f73d1a738f None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:27.942950 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d87112-88c7-4be4-8185-6244d6de3c76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:27.943540 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d87112-88c7-4be4-8185-6244d6de3c76 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:27.943817 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d87112-88c7-4be4-8185-6244d6de3c76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:27.944088 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d87112-88c7-4be4-8185-6244d6de3c76 admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:27.944709 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d87112-88c7-4be4-8185-6244d6de3c76 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:27.946624 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d87112-88c7-4be4-8185-6244d6de3c76 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:27.977381 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1278/2543] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:40:27 2022] POST /identity/v3/projects => generated 365 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:40:27.979867 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a4ed7e-f9d2-42d7-88b7-b69af81696eb admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:27.989564 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a4ed7e-f9d2-42d7-88b7-b69af81696eb admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:28.015486 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:40:28.047109 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770afdf2-5d55-4127-8f28-5595ed62f0f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:28.047702 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770afdf2-5d55-4127-8f28-5595ed62f0f8 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:28.047976 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770afdf2-5d55-4127-8f28-5595ed62f0f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:28.048244 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770afdf2-5d55-4127-8f28-5595ed62f0f8 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:28.048853 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770afdf2-5d55-4127-8f28-5595ed62f0f8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:28.050790 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770afdf2-5d55-4127-8f28-5595ed62f0f8 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:28.072628 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1266/2544] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:40:27 2022] POST /identity/v3/users => generated 283 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:40:28.076718 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d87112-88c7-4be4-8185-6244d6de3c76 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:28.086850 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d87112-88c7-4be4-8185-6244d6de3c76 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:28.092486 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585b5d61-31e1-45c1-a85d-2210cbded55d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:28.093091 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585b5d61-31e1-45c1-a85d-2210cbded55d admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:28.093376 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585b5d61-31e1-45c1-a85d-2210cbded55d admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:28.093641 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585b5d61-31e1-45c1-a85d-2210cbded55d admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:28.094269 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-585b5d61-31e1-45c1-a85d-2210cbded55d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:28.095845 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-585b5d61-31e1-45c1-a85d-2210cbded55d admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:28.106591 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1279/2545] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:40:28 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:40:28.110557 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770afdf2-5d55-4127-8f28-5595ed62f0f8 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:28.120612 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770afdf2-5d55-4127-8f28-5595ed62f0f8 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:28.125189 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c33911-b9c5-4ccd-9a34-3a7a197bf198 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:28.125775 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c33911-b9c5-4ccd-9a34-3a7a197bf198 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:28.126052 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c33911-b9c5-4ccd-9a34-3a7a197bf198 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:28.126321 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c33911-b9c5-4ccd-9a34-3a7a197bf198 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:28.126952 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c33911-b9c5-4ccd-9a34-3a7a197bf198 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:28.128560 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c33911-b9c5-4ccd-9a34-3a7a197bf198 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:28.139139 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1267/2546] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:40:28 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:40:28.143120 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585b5d61-31e1-45c1-a85d-2210cbded55d admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:28.154119 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585b5d61-31e1-45c1-a85d-2210cbded55d admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:28.158673 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9daaa97f-6317-44c2-9f1a-4c7125785270 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:28.159311 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9daaa97f-6317-44c2-9f1a-4c7125785270 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:28.159601 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9daaa97f-6317-44c2-9f1a-4c7125785270 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:28.159850 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9daaa97f-6317-44c2-9f1a-4c7125785270 admin admin] PATH_INFO: `/v3/projects/f2f988f14a3f4a0ba6ccda1a3d751d1a/users/f2baeca472e645d2aa1ef223294aa860/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:28.174978 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9daaa97f-6317-44c2-9f1a-4c7125785270 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f2f988f14a3f4a0ba6ccda1a3d751d1a, user_id=f2baeca472e645d2aa1ef223294aa860, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:28.177633 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9daaa97f-6317-44c2-9f1a-4c7125785270 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:28.190897 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1280/2547] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:40:28 2022] PUT /identity/v3/projects/f2f988f14a3f4a0ba6ccda1a3d751d1a/users/f2baeca472e645d2aa1ef223294aa860/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:40:28.201032 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40c758f-683d-4b11-ad59-fe3034876136 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:28.201340 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40c758f-683d-4b11-ad59-fe3034876136 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:28.201611 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40c758f-683d-4b11-ad59-fe3034876136 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:28.229706 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-c40c758f-683d-4b11-ad59-fe3034876136 None None] MFA Rules not processed for user `f2baeca472e645d2aa1ef223294aa860`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:40:28.230924 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-c40c758f-683d-4b11-ad59-fe3034876136 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:40:28.293728 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1268/2548] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:40:28 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:40:28.298171 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9daaa97f-6317-44c2-9f1a-4c7125785270 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:28.359080 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9daaa97f-6317-44c2-9f1a-4c7125785270 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:28.363817 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d28655-5cce-49af-8669-de3bddea0acb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:28.364420 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d28655-5cce-49af-8669-de3bddea0acb admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:28.364699 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d28655-5cce-49af-8669-de3bddea0acb admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:28.364969 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d28655-5cce-49af-8669-de3bddea0acb admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:28.365632 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d28655-5cce-49af-8669-de3bddea0acb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:28.368090 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d28655-5cce-49af-8669-de3bddea0acb admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:28.394189 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1281/2549] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:40:28 2022] POST /identity/v3/projects => generated 365 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:40:28.398249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40c758f-683d-4b11-ad59-fe3034876136 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:28.409250 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40c758f-683d-4b11-ad59-fe3034876136 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:28.466249 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eac547a-0a12-4931-b9c7-8707852f7c9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:28.466771 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eac547a-0a12-4931-b9c7-8707852f7c9a admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:28.467048 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eac547a-0a12-4931-b9c7-8707852f7c9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:28.467322 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eac547a-0a12-4931-b9c7-8707852f7c9a admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:28.467946 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eac547a-0a12-4931-b9c7-8707852f7c9a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:28.469880 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eac547a-0a12-4931-b9c7-8707852f7c9a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:28.491372 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1269/2550] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:40:28 2022] POST /identity/v3/users => generated 289 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:40:28.495553 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d28655-5cce-49af-8669-de3bddea0acb admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:28.505841 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d28655-5cce-49af-8669-de3bddea0acb admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:28.510995 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31254f88-b091-4e85-bcc5-f1e4eef78909 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:28.511596 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31254f88-b091-4e85-bcc5-f1e4eef78909 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:28.511875 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31254f88-b091-4e85-bcc5-f1e4eef78909 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:28.512151 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31254f88-b091-4e85-bcc5-f1e4eef78909 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:28.512783 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31254f88-b091-4e85-bcc5-f1e4eef78909 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:28.514417 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31254f88-b091-4e85-bcc5-f1e4eef78909 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:28.525975 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1282/2551] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:40:28 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:40:28.529837 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eac547a-0a12-4931-b9c7-8707852f7c9a admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:28.539612 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eac547a-0a12-4931-b9c7-8707852f7c9a admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:28.544414 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d34fd3b-f582-404b-83bd-54879991ae31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:28.545052 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d34fd3b-f582-404b-83bd-54879991ae31 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:28.545390 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d34fd3b-f582-404b-83bd-54879991ae31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:28.545656 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d34fd3b-f582-404b-83bd-54879991ae31 admin admin] PATH_INFO: `/v3/projects/2dfe84db0ec34e858a7799dea541aab8/users/06d8cc12e38243db952431d2ca1f7178/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:28.561362 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d34fd3b-f582-404b-83bd-54879991ae31 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2dfe84db0ec34e858a7799dea541aab8, user_id=06d8cc12e38243db952431d2ca1f7178, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:28.564022 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d34fd3b-f582-404b-83bd-54879991ae31 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:28.576795 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1270/2552] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:40:28 2022] PUT /identity/v3/projects/2dfe84db0ec34e858a7799dea541aab8/users/06d8cc12e38243db952431d2ca1f7178/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:40:28.580690 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31254f88-b091-4e85-bcc5-f1e4eef78909 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:28.591254 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31254f88-b091-4e85-bcc5-f1e4eef78909 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:28.611237 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:40:28.652834 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05532f4d-6ad1-4c14-89c6-124af3569d41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:28.653475 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05532f4d-6ad1-4c14-89c6-124af3569d41 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:28.653755 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05532f4d-6ad1-4c14-89c6-124af3569d41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:28.654081 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05532f4d-6ad1-4c14-89c6-124af3569d41 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:28.654723 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05532f4d-6ad1-4c14-89c6-124af3569d41 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:28.656358 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05532f4d-6ad1-4c14-89c6-124af3569d41 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:28.666109 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1283/2553] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:40:28 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:40:28.670241 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d34fd3b-f582-404b-83bd-54879991ae31 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:28.675673 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:40:28.682232 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d34fd3b-f582-404b-83bd-54879991ae31 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:28.687342 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae4c455-bf52-4b74-aeff-b6ad705bde12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:28.687997 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae4c455-bf52-4b74-aeff-b6ad705bde12 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:28.688276 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae4c455-bf52-4b74-aeff-b6ad705bde12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:28.688548 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae4c455-bf52-4b74-aeff-b6ad705bde12 admin admin] PATH_INFO: `/v3/projects/2dfe84db0ec34e858a7799dea541aab8/users/06d8cc12e38243db952431d2ca1f7178/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:28.690485 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae4c455-bf52-4b74-aeff-b6ad705bde12 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2dfe84db0ec34e858a7799dea541aab8, user_id=06d8cc12e38243db952431d2ca1f7178, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:28.693188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae4c455-bf52-4b74-aeff-b6ad705bde12 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:28.703852 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1271/2554] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:40:28 2022] PUT /identity/v3/projects/2dfe84db0ec34e858a7799dea541aab8/users/06d8cc12e38243db952431d2ca1f7178/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:40:28.715993 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4354459a-38bb-4727-ad6b-0086add7ccbf None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:28.716553 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4354459a-38bb-4727-ad6b-0086add7ccbf None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:28.716553 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4354459a-38bb-4727-ad6b-0086add7ccbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:28.748385 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-4354459a-38bb-4727-ad6b-0086add7ccbf None None] MFA Rules not processed for user `06d8cc12e38243db952431d2ca1f7178`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:40:28.749693 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-4354459a-38bb-4727-ad6b-0086add7ccbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:40:28.839500 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1284/2555] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:40:28 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:40:28.853928 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae4c455-bf52-4b74-aeff-b6ad705bde12 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:28.855729 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-4ae4c455-bf52-4b74-aeff-b6ad705bde12 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:40:28.923544 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae4c455-bf52-4b74-aeff-b6ad705bde12 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:28.927381 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d1b0e7-da04-412d-842f-80cf781dcc5b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:28.928305 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d1b0e7-da04-412d-842f-80cf781dcc5b service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:28.928813 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d1b0e7-da04-412d-842f-80cf781dcc5b service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:28.929354 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d1b0e7-da04-412d-842f-80cf781dcc5b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:28.941189 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d1b0e7-da04-412d-842f-80cf781dcc5b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:28.942625 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d1b0e7-da04-412d-842f-80cf781dcc5b service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:28.953947 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1272/2556] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:40:28 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:40:29.048243 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4354459a-38bb-4727-ad6b-0086add7ccbf None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:29.059947 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4354459a-38bb-4727-ad6b-0086add7ccbf None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:29.064135 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab0596b-a4b2-469f-8fe7-7d73740bf1be service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:29.064738 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab0596b-a4b2-469f-8fe7-7d73740bf1be service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:29.065017 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab0596b-a4b2-469f-8fe7-7d73740bf1be service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:29.065320 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab0596b-a4b2-469f-8fe7-7d73740bf1be service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:29.134155 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab0596b-a4b2-469f-8fe7-7d73740bf1be service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:29.136292 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab0596b-a4b2-469f-8fe7-7d73740bf1be service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:29.145921 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1285/2557] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:40:29 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:40:38.948942 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=30,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:40:39.141319 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=80 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:40:39.571195 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d1b0e7-da04-412d-842f-80cf781dcc5b service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:39.635274 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d1b0e7-da04-412d-842f-80cf781dcc5b service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:39.639830 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7847f6f2-027e-4e8c-af30-384f1e8c6282 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:39.640450 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7847f6f2-027e-4e8c-af30-384f1e8c6282 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:39.640728 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7847f6f2-027e-4e8c-af30-384f1e8c6282 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:39.640996 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7847f6f2-027e-4e8c-af30-384f1e8c6282 admin admin] PATH_INFO: `/v3/users/f2baeca472e645d2aa1ef223294aa860` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:39.642137 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7847f6f2-027e-4e8c-af30-384f1e8c6282 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f2baeca472e645d2aa1ef223294aa860)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:39.644076 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7847f6f2-027e-4e8c-af30-384f1e8c6282 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:39.690239 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-7847f6f2-027e-4e8c-af30-384f1e8c6282 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f2baeca472e645d2aa1ef223294aa860'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:39.698193 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-7847f6f2-027e-4e8c-af30-384f1e8c6282 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f2baeca472e645d2aa1ef223294aa860'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:39.698902 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-7847f6f2-027e-4e8c-af30-384f1e8c6282 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f2baeca472e645d2aa1ef223294aa860'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:39.710743 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-7847f6f2-027e-4e8c-af30-384f1e8c6282 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f2baeca472e645d2aa1ef223294aa860'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:39.718133 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-7847f6f2-027e-4e8c-af30-384f1e8c6282 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f2baeca472e645d2aa1ef223294aa860'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:39.723986 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1273/2558] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:40:39 2022] DELETE /identity/v3/users/f2baeca472e645d2aa1ef223294aa860 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:40:39.729360 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba33e5a-d3d4-42dd-831b-7ea0d15918f8 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:39.729983 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba33e5a-d3d4-42dd-831b-7ea0d15918f8 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:39.730253 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba33e5a-d3d4-42dd-831b-7ea0d15918f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:39.750880 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-fba33e5a-d3d4-42dd-831b-7ea0d15918f8 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:40:39.752066 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-fba33e5a-d3d4-42dd-831b-7ea0d15918f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:40:39.812964 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1286/2559] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:40:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:40:39.822614 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7847f6f2-027e-4e8c-af30-384f1e8c6282 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:39.824151 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-7847f6f2-027e-4e8c-af30-384f1e8c6282 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:40:39.880853 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7847f6f2-027e-4e8c-af30-384f1e8c6282 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:39.884513 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29195d6-0431-49c6-836b-77ea89716b48 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:39.885207 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29195d6-0431-49c6-836b-77ea89716b48 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:39.885487 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29195d6-0431-49c6-836b-77ea89716b48 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:39.885758 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29195d6-0431-49c6-836b-77ea89716b48 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:39.895647 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d29195d6-0431-49c6-836b-77ea89716b48 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:39.897101 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d29195d6-0431-49c6-836b-77ea89716b48 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:39.906691 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1274/2560] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:40:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:40:40.171957 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba33e5a-d3d4-42dd-831b-7ea0d15918f8 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:40.173605 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-fba33e5a-d3d4-42dd-831b-7ea0d15918f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:40:40.187219 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba33e5a-d3d4-42dd-831b-7ea0d15918f8 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:40.192218 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242df0ad-52c4-4210-98b1-f5e7abffadf6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:40.192846 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242df0ad-52c4-4210-98b1-f5e7abffadf6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:40.193148 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242df0ad-52c4-4210-98b1-f5e7abffadf6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:40.193419 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242df0ad-52c4-4210-98b1-f5e7abffadf6 admin admin] PATH_INFO: `/v3/projects/f2f988f14a3f4a0ba6ccda1a3d751d1a` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:40.194531 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-242df0ad-52c4-4210-98b1-f5e7abffadf6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f2f988f14a3f4a0ba6ccda1a3d751d1a)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:40.196481 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-242df0ad-52c4-4210-98b1-f5e7abffadf6 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:40.247023 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-242df0ad-52c4-4210-98b1-f5e7abffadf6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f2f988f14a3f4a0ba6ccda1a3d751d1a', 'request_id': 'req-242df0ad-52c4-4210-98b1-f5e7abffadf6'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:40.254853 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-242df0ad-52c4-4210-98b1-f5e7abffadf6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f2f988f14a3f4a0ba6ccda1a3d751d1a', 'request_id': 'req-242df0ad-52c4-4210-98b1-f5e7abffadf6'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:40.261266 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-242df0ad-52c4-4210-98b1-f5e7abffadf6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f2f988f14a3f4a0ba6ccda1a3d751d1a', 'request_id': 'req-242df0ad-52c4-4210-98b1-f5e7abffadf6'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:40.271732 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-242df0ad-52c4-4210-98b1-f5e7abffadf6 admin admin] The token cache is being invalidate because project f2f988f14a3f4a0ba6ccda1a3d751d1a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:40:40.272019 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-242df0ad-52c4-4210-98b1-f5e7abffadf6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:40.272639 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-242df0ad-52c4-4210-98b1-f5e7abffadf6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:40.274156 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1287/2561] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:40:40 2022] DELETE /identity/v3/projects/f2f988f14a3f4a0ba6ccda1a3d751d1a => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:40:40.278135 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29195d6-0431-49c6-836b-77ea89716b48 service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:40.287738 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29195d6-0431-49c6-836b-77ea89716b48 service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:40.305976 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:40:40.345188 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6825296b-3b78-4675-b9be-dd996fd6e9b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:40.345799 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6825296b-3b78-4675-b9be-dd996fd6e9b6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:40.346076 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6825296b-3b78-4675-b9be-dd996fd6e9b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:40.346348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6825296b-3b78-4675-b9be-dd996fd6e9b6 admin admin] PATH_INFO: `/v3/users/06d8cc12e38243db952431d2ca1f7178` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:40.347461 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6825296b-3b78-4675-b9be-dd996fd6e9b6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=06d8cc12e38243db952431d2ca1f7178)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:40.349399 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6825296b-3b78-4675-b9be-dd996fd6e9b6 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:40.394516 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6825296b-3b78-4675-b9be-dd996fd6e9b6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '06d8cc12e38243db952431d2ca1f7178'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:40.401338 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6825296b-3b78-4675-b9be-dd996fd6e9b6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '06d8cc12e38243db952431d2ca1f7178'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:40.402133 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6825296b-3b78-4675-b9be-dd996fd6e9b6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '06d8cc12e38243db952431d2ca1f7178'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:40.413853 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6825296b-3b78-4675-b9be-dd996fd6e9b6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '06d8cc12e38243db952431d2ca1f7178'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:40.421264 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6825296b-3b78-4675-b9be-dd996fd6e9b6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '06d8cc12e38243db952431d2ca1f7178'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:40.427063 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1275/2562] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:40:40 2022] DELETE /identity/v3/users/06d8cc12e38243db952431d2ca1f7178 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:40:40.701646 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242df0ad-52c4-4210-98b1-f5e7abffadf6 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:40.703074 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-242df0ad-52c4-4210-98b1-f5e7abffadf6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:40:40.716808 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242df0ad-52c4-4210-98b1-f5e7abffadf6 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:40.773359 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-509a67cf-7cf2-493e-8250-4007d52f1dee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:40.773989 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509a67cf-7cf2-493e-8250-4007d52f1dee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:40.774267 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509a67cf-7cf2-493e-8250-4007d52f1dee admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:40.774539 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509a67cf-7cf2-493e-8250-4007d52f1dee admin admin] PATH_INFO: `/v3/projects/2dfe84db0ec34e858a7799dea541aab8` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:40.775698 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-509a67cf-7cf2-493e-8250-4007d52f1dee admin admin] RBAC: Authorizing `identity:delete_project(project_id=2dfe84db0ec34e858a7799dea541aab8)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:40.778110 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-509a67cf-7cf2-493e-8250-4007d52f1dee admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:40.810744 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:40:40.829225 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-509a67cf-7cf2-493e-8250-4007d52f1dee admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2dfe84db0ec34e858a7799dea541aab8', 'request_id': 'req-509a67cf-7cf2-493e-8250-4007d52f1dee'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:40.836297 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-509a67cf-7cf2-493e-8250-4007d52f1dee admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2dfe84db0ec34e858a7799dea541aab8', 'request_id': 'req-509a67cf-7cf2-493e-8250-4007d52f1dee'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:40.843198 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-509a67cf-7cf2-493e-8250-4007d52f1dee admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2dfe84db0ec34e858a7799dea541aab8', 'request_id': 'req-509a67cf-7cf2-493e-8250-4007d52f1dee'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:40.853766 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-509a67cf-7cf2-493e-8250-4007d52f1dee admin admin] The token cache is being invalidate because project 2dfe84db0ec34e858a7799dea541aab8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:40:40.854050 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-509a67cf-7cf2-493e-8250-4007d52f1dee admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:40.854688 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-509a67cf-7cf2-493e-8250-4007d52f1dee admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:40.856176 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1288/2563] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:40:40 2022] DELETE /identity/v3/projects/2dfe84db0ec34e858a7799dea541aab8 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:40:40.861051 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1686858-aa82-422d-b63a-8e6ff4556988 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:40.861354 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1686858-aa82-422d-b63a-8e6ff4556988 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:40.861626 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1686858-aa82-422d-b63a-8e6ff4556988 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:40.880710 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-f1686858-aa82-422d-b63a-8e6ff4556988 None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:40:40.881931 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f1686858-aa82-422d-b63a-8e6ff4556988 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:40:40.939459 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1276/2564] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:40:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:40:40.947233 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35537bdb-0c87-453f-8244-fd5887a4df1f None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:40.947515 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35537bdb-0c87-453f-8244-fd5887a4df1f None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:40.947785 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35537bdb-0c87-453f-8244-fd5887a4df1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:40.968233 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-35537bdb-0c87-453f-8244-fd5887a4df1f None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:40:40.969440 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-35537bdb-0c87-453f-8244-fd5887a4df1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:40:40.977158 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1289/2565] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:40:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:40:40.981415 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1686858-aa82-422d-b63a-8e6ff4556988 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:40.990781 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1686858-aa82-422d-b63a-8e6ff4556988 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:40.995276 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bdfbafe-368f-45aa-8330-3ee2d0ee6078 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:40.996061 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdfbafe-368f-45aa-8330-3ee2d0ee6078 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:40.996341 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdfbafe-368f-45aa-8330-3ee2d0ee6078 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:40.996608 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdfbafe-368f-45aa-8330-3ee2d0ee6078 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:40.997284 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bdfbafe-368f-45aa-8330-3ee2d0ee6078 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:40:40.997585 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bdfbafe-368f-45aa-8330-3ee2d0ee6078 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:40.999158 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bdfbafe-368f-45aa-8330-3ee2d0ee6078 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:41.010347 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1277/2566] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:40:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:40:41.015127 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35537bdb-0c87-453f-8244-fd5887a4df1f None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:41.024187 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35537bdb-0c87-453f-8244-fd5887a4df1f None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:41.029693 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15be52fe-cab3-44c4-904d-3818cf3394bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:41.030223 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15be52fe-cab3-44c4-904d-3818cf3394bb admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:41.030501 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15be52fe-cab3-44c4-904d-3818cf3394bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:41.030782 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15be52fe-cab3-44c4-904d-3818cf3394bb admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:41.031392 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15be52fe-cab3-44c4-904d-3818cf3394bb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:41.033281 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15be52fe-cab3-44c4-904d-3818cf3394bb admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:41.059868 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1290/2567] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:40:41 2022] POST /identity/v3/projects => generated 383 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:40:41.063925 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bdfbafe-368f-45aa-8330-3ee2d0ee6078 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:41.074337 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bdfbafe-368f-45aa-8330-3ee2d0ee6078 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:41.099796 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:40:41.132042 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396fb8b8-21db-4e88-a591-11faeeed7c58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:41.132637 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396fb8b8-21db-4e88-a591-11faeeed7c58 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:41.132917 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396fb8b8-21db-4e88-a591-11faeeed7c58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:41.133226 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396fb8b8-21db-4e88-a591-11faeeed7c58 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:41.133847 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-396fb8b8-21db-4e88-a591-11faeeed7c58 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:41.135743 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-396fb8b8-21db-4e88-a591-11faeeed7c58 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:41.157571 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1278/2568] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:40:41 2022] POST /identity/v3/users => generated 292 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:40:41.161657 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15be52fe-cab3-44c4-904d-3818cf3394bb admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:41.172481 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15be52fe-cab3-44c4-904d-3818cf3394bb admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:41.177140 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119cbcc7-aeb8-4f69-81b5-f50215123608 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:41.177728 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119cbcc7-aeb8-4f69-81b5-f50215123608 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:41.178011 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119cbcc7-aeb8-4f69-81b5-f50215123608 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:41.178283 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119cbcc7-aeb8-4f69-81b5-f50215123608 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:41.178914 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-119cbcc7-aeb8-4f69-81b5-f50215123608 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:41.180500 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-119cbcc7-aeb8-4f69-81b5-f50215123608 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:41.190743 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1291/2569] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:40:41 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:40:41.194718 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396fb8b8-21db-4e88-a591-11faeeed7c58 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:41.204640 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396fb8b8-21db-4e88-a591-11faeeed7c58 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:41.209328 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9efd9158-796b-4026-bfba-0515fa1ceb8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:41.209918 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9efd9158-796b-4026-bfba-0515fa1ceb8a admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:41.210197 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9efd9158-796b-4026-bfba-0515fa1ceb8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:41.210466 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9efd9158-796b-4026-bfba-0515fa1ceb8a admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:41.211103 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9efd9158-796b-4026-bfba-0515fa1ceb8a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:41.212685 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9efd9158-796b-4026-bfba-0515fa1ceb8a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:41.224907 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1279/2570] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:40:41 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:40:41.229011 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119cbcc7-aeb8-4f69-81b5-f50215123608 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:41.239789 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119cbcc7-aeb8-4f69-81b5-f50215123608 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:41.244360 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746920ac-0a95-4286-94b1-01fa8e44bd78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:41.244997 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746920ac-0a95-4286-94b1-01fa8e44bd78 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:41.245316 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746920ac-0a95-4286-94b1-01fa8e44bd78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:41.245596 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746920ac-0a95-4286-94b1-01fa8e44bd78 admin admin] PATH_INFO: `/v3/projects/24e7f424102345249093693cd496a930/users/116f1fa6f71546d2be27e9d6b8059f0e/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:41.259836 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746920ac-0a95-4286-94b1-01fa8e44bd78 admin admin] RBAC: Authorizing `identity:create_grant(project_id=24e7f424102345249093693cd496a930, user_id=116f1fa6f71546d2be27e9d6b8059f0e, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:41.262510 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746920ac-0a95-4286-94b1-01fa8e44bd78 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:41.275605 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1292/2571] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:40:41 2022] PUT /identity/v3/projects/24e7f424102345249093693cd496a930/users/116f1fa6f71546d2be27e9d6b8059f0e/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:40:41.287552 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea86034f-0077-4956-be0d-44b3897c8f4a None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:41.288037 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea86034f-0077-4956-be0d-44b3897c8f4a None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:41.288037 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea86034f-0077-4956-be0d-44b3897c8f4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:41.315820 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-ea86034f-0077-4956-be0d-44b3897c8f4a None None] MFA Rules not processed for user `116f1fa6f71546d2be27e9d6b8059f0e`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:40:41.317020 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ea86034f-0077-4956-be0d-44b3897c8f4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:40:41.379716 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1280/2572] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:40:41 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:40:41.384186 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746920ac-0a95-4286-94b1-01fa8e44bd78 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:41.445449 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746920ac-0a95-4286-94b1-01fa8e44bd78 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:41.450589 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d7df362-f84e-4e86-9dbd-a060e2c69c2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:41.451177 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7df362-f84e-4e86-9dbd-a060e2c69c2b admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:41.451447 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7df362-f84e-4e86-9dbd-a060e2c69c2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:41.451710 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7df362-f84e-4e86-9dbd-a060e2c69c2b admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:41.452332 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d7df362-f84e-4e86-9dbd-a060e2c69c2b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:41.454303 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d7df362-f84e-4e86-9dbd-a060e2c69c2b admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:41.481663 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1293/2573] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:40:41 2022] POST /identity/v3/projects => generated 381 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:40:41.485747 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea86034f-0077-4956-be0d-44b3897c8f4a None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:41.496518 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea86034f-0077-4956-be0d-44b3897c8f4a None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:41.552189 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a6acc6-4609-4ee3-b1cf-c655d48501a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:41.552690 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a6acc6-4609-4ee3-b1cf-c655d48501a7 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:41.552964 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a6acc6-4609-4ee3-b1cf-c655d48501a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:41.553262 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a6acc6-4609-4ee3-b1cf-c655d48501a7 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:41.553875 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a6acc6-4609-4ee3-b1cf-c655d48501a7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:41.555763 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a6acc6-4609-4ee3-b1cf-c655d48501a7 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:41.577550 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1281/2574] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:40:41 2022] POST /identity/v3/users => generated 297 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:40:41.581509 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d7df362-f84e-4e86-9dbd-a060e2c69c2b admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:41.592171 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d7df362-f84e-4e86-9dbd-a060e2c69c2b admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:41.596668 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587a8b4e-c1be-4b30-a301-766374502e70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:41.597283 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587a8b4e-c1be-4b30-a301-766374502e70 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:41.597560 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587a8b4e-c1be-4b30-a301-766374502e70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:41.597826 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587a8b4e-c1be-4b30-a301-766374502e70 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:41.598626 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-587a8b4e-c1be-4b30-a301-766374502e70 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:41.600066 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-587a8b4e-c1be-4b30-a301-766374502e70 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:41.611170 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1294/2575] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:40:41 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:40:41.615232 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a6acc6-4609-4ee3-b1cf-c655d48501a7 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:41.625474 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a6acc6-4609-4ee3-b1cf-c655d48501a7 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:41.630243 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534741d1-b73a-42de-b6fc-4b07fc97f703 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:41.630880 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534741d1-b73a-42de-b6fc-4b07fc97f703 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:41.631155 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534741d1-b73a-42de-b6fc-4b07fc97f703 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:41.631425 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534741d1-b73a-42de-b6fc-4b07fc97f703 admin admin] PATH_INFO: `/v3/projects/3627d7329b434d4684fda14b7835996e/users/dce5439a065c42dcba67d9d37b3de096/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:41.646750 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534741d1-b73a-42de-b6fc-4b07fc97f703 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3627d7329b434d4684fda14b7835996e, user_id=dce5439a065c42dcba67d9d37b3de096, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:41.649420 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534741d1-b73a-42de-b6fc-4b07fc97f703 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:41.663737 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1282/2576] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:40:41 2022] PUT /identity/v3/projects/3627d7329b434d4684fda14b7835996e/users/dce5439a065c42dcba67d9d37b3de096/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 27 01:40:41.667787 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587a8b4e-c1be-4b30-a301-766374502e70 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:41.677830 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587a8b4e-c1be-4b30-a301-766374502e70 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:41.697478 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:40:41.735875 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1484ef-697e-491a-a226-54e2ad4672db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:41.736469 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1484ef-697e-491a-a226-54e2ad4672db admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:41.736745 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1484ef-697e-491a-a226-54e2ad4672db admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:41.737015 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1484ef-697e-491a-a226-54e2ad4672db admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:41.737671 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d1484ef-697e-491a-a226-54e2ad4672db admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:41.739237 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d1484ef-697e-491a-a226-54e2ad4672db admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:41.748727 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1295/2577] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:40:41 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:40:41.752982 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534741d1-b73a-42de-b6fc-4b07fc97f703 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:41.758372 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:40:41.764464 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534741d1-b73a-42de-b6fc-4b07fc97f703 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:41.770561 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a72844-9d62-4b22-8a71-1da41896894f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:41.771257 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a72844-9d62-4b22-8a71-1da41896894f admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:41.771539 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a72844-9d62-4b22-8a71-1da41896894f admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:41.771833 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a72844-9d62-4b22-8a71-1da41896894f admin admin] PATH_INFO: `/v3/projects/3627d7329b434d4684fda14b7835996e/users/dce5439a065c42dcba67d9d37b3de096/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:41.773771 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a72844-9d62-4b22-8a71-1da41896894f admin admin] RBAC: Authorizing `identity:create_grant(project_id=3627d7329b434d4684fda14b7835996e, user_id=dce5439a065c42dcba67d9d37b3de096, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:41.776541 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a72844-9d62-4b22-8a71-1da41896894f admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:41.788007 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1283/2578] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:40:41 2022] PUT /identity/v3/projects/3627d7329b434d4684fda14b7835996e/users/dce5439a065c42dcba67d9d37b3de096/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:40:41.798275 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992708bd-75b2-403c-bcd9-05b07a7f67cd None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:41.798589 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992708bd-75b2-403c-bcd9-05b07a7f67cd None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:41.798862 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992708bd-75b2-403c-bcd9-05b07a7f67cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:41.828272 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-992708bd-75b2-403c-bcd9-05b07a7f67cd None None] MFA Rules not processed for user `dce5439a065c42dcba67d9d37b3de096`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:40:41.829596 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-992708bd-75b2-403c-bcd9-05b07a7f67cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:40:41.899431 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1296/2579] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:40:41 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:40:41.911601 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a72844-9d62-4b22-8a71-1da41896894f admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:41.913382 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-85a72844-9d62-4b22-8a71-1da41896894f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:40:41.972047 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a72844-9d62-4b22-8a71-1da41896894f admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:41.975647 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2ad10a-ee60-4cda-b228-77c122657154 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:41.976241 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2ad10a-ee60-4cda-b228-77c122657154 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:41.976512 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2ad10a-ee60-4cda-b228-77c122657154 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:41.976778 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2ad10a-ee60-4cda-b228-77c122657154 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:41.986321 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2ad10a-ee60-4cda-b228-77c122657154 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:41.987728 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2ad10a-ee60-4cda-b228-77c122657154 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:41.999052 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1284/2580] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:40:41 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:40:42.073249 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992708bd-75b2-403c-bcd9-05b07a7f67cd None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:42.083556 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992708bd-75b2-403c-bcd9-05b07a7f67cd None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:42.087143 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e37e86-45f4-4b54-b09d-c72b14042022 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:42.087739 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e37e86-45f4-4b54-b09d-c72b14042022 service cinder] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:42.088016 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e37e86-45f4-4b54-b09d-c72b14042022 service cinder] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:42.088285 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e37e86-45f4-4b54-b09d-c72b14042022 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:42.144659 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7e37e86-45f4-4b54-b09d-c72b14042022 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:42.146125 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7e37e86-45f4-4b54-b09d-c72b14042022 service cinder] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:42.155633 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1297/2581] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:40:42 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:40:42.416710 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2ad10a-ee60-4cda-b228-77c122657154 service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:42.485952 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2ad10a-ee60-4cda-b228-77c122657154 service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:42.490737 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caba0a9a-38ed-4d7c-b794-8aeea11ca8eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:42.491354 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caba0a9a-38ed-4d7c-b794-8aeea11ca8eb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:42.491628 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caba0a9a-38ed-4d7c-b794-8aeea11ca8eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:42.491901 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caba0a9a-38ed-4d7c-b794-8aeea11ca8eb admin admin] PATH_INFO: `/v3/users/116f1fa6f71546d2be27e9d6b8059f0e` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:42.492999 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caba0a9a-38ed-4d7c-b794-8aeea11ca8eb admin admin] RBAC: Authorizing `identity:delete_user(user_id=116f1fa6f71546d2be27e9d6b8059f0e)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:42.494934 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caba0a9a-38ed-4d7c-b794-8aeea11ca8eb admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:42.540268 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-caba0a9a-38ed-4d7c-b794-8aeea11ca8eb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '116f1fa6f71546d2be27e9d6b8059f0e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:42.548237 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-caba0a9a-38ed-4d7c-b794-8aeea11ca8eb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '116f1fa6f71546d2be27e9d6b8059f0e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:42.548887 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-caba0a9a-38ed-4d7c-b794-8aeea11ca8eb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '116f1fa6f71546d2be27e9d6b8059f0e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:42.562131 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-caba0a9a-38ed-4d7c-b794-8aeea11ca8eb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '116f1fa6f71546d2be27e9d6b8059f0e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:42.568880 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-caba0a9a-38ed-4d7c-b794-8aeea11ca8eb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '116f1fa6f71546d2be27e9d6b8059f0e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:42.574748 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1285/2582] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:40:42 2022] DELETE /identity/v3/users/116f1fa6f71546d2be27e9d6b8059f0e => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:40:42.579223 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50eca9d2-3f66-439f-9890-706fe4f1c0cc None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:42.579712 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50eca9d2-3f66-439f-9890-706fe4f1c0cc None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:42.579836 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50eca9d2-3f66-439f-9890-706fe4f1c0cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:42.598661 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-50eca9d2-3f66-439f-9890-706fe4f1c0cc None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:40:42.599784 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-50eca9d2-3f66-439f-9890-706fe4f1c0cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:40:42.639930 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=100 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:40:42.658770 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1298/2583] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:40:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:40:42.669206 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caba0a9a-38ed-4d7c-b794-8aeea11ca8eb admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:42.670620 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-caba0a9a-38ed-4d7c-b794-8aeea11ca8eb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:40:42.684836 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:40:42.729580 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caba0a9a-38ed-4d7c-b794-8aeea11ca8eb admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:42.733276 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbbd911d-cdff-4301-84fc-6e2bdc30eeea service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:42.733875 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbd911d-cdff-4301-84fc-6e2bdc30eeea service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:42.734151 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbd911d-cdff-4301-84fc-6e2bdc30eeea service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:42.734422 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbd911d-cdff-4301-84fc-6e2bdc30eeea service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:42.743148 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbbd911d-cdff-4301-84fc-6e2bdc30eeea service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:42.744561 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbbd911d-cdff-4301-84fc-6e2bdc30eeea service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:42.754462 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1286/2584] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:40:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:40:43.017321 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50eca9d2-3f66-439f-9890-706fe4f1c0cc None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:43.018728 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-50eca9d2-3f66-439f-9890-706fe4f1c0cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:40:43.032119 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50eca9d2-3f66-439f-9890-706fe4f1c0cc None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:43.036713 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22054ba0-7427-4098-8884-bfab4823653d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:43.037376 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22054ba0-7427-4098-8884-bfab4823653d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:43.037655 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22054ba0-7427-4098-8884-bfab4823653d admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:43.037936 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22054ba0-7427-4098-8884-bfab4823653d admin admin] PATH_INFO: `/v3/projects/24e7f424102345249093693cd496a930` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:43.039039 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22054ba0-7427-4098-8884-bfab4823653d admin admin] RBAC: Authorizing `identity:delete_project(project_id=24e7f424102345249093693cd496a930)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:43.040940 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22054ba0-7427-4098-8884-bfab4823653d admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:43.089070 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-22054ba0-7427-4098-8884-bfab4823653d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '24e7f424102345249093693cd496a930', 'request_id': 'req-22054ba0-7427-4098-8884-bfab4823653d'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:43.096627 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-22054ba0-7427-4098-8884-bfab4823653d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '24e7f424102345249093693cd496a930', 'request_id': 'req-22054ba0-7427-4098-8884-bfab4823653d'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:43.102836 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-22054ba0-7427-4098-8884-bfab4823653d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '24e7f424102345249093693cd496a930', 'request_id': 'req-22054ba0-7427-4098-8884-bfab4823653d'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:43.114028 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-22054ba0-7427-4098-8884-bfab4823653d admin admin] The token cache is being invalidate because project 24e7f424102345249093693cd496a930 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:40:43.114305 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-22054ba0-7427-4098-8884-bfab4823653d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:43.114931 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-22054ba0-7427-4098-8884-bfab4823653d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:43.116384 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1299/2585] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:40:43 2022] DELETE /identity/v3/projects/24e7f424102345249093693cd496a930 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:40:43.121237 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbbd911d-cdff-4301-84fc-6e2bdc30eeea service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:43.131199 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbbd911d-cdff-4301-84fc-6e2bdc30eeea service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:43.188222 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8ba415-1b2a-4332-8643-3915b96d7c1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:43.188837 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8ba415-1b2a-4332-8643-3915b96d7c1e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:43.189146 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8ba415-1b2a-4332-8643-3915b96d7c1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:43.189420 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8ba415-1b2a-4332-8643-3915b96d7c1e admin admin] PATH_INFO: `/v3/users/dce5439a065c42dcba67d9d37b3de096` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:43.190519 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb8ba415-1b2a-4332-8643-3915b96d7c1e admin admin] RBAC: Authorizing `identity:delete_user(user_id=dce5439a065c42dcba67d9d37b3de096)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:43.192456 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb8ba415-1b2a-4332-8643-3915b96d7c1e admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:43.237439 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-cb8ba415-1b2a-4332-8643-3915b96d7c1e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dce5439a065c42dcba67d9d37b3de096'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:43.244267 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-cb8ba415-1b2a-4332-8643-3915b96d7c1e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dce5439a065c42dcba67d9d37b3de096'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:43.244866 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-cb8ba415-1b2a-4332-8643-3915b96d7c1e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dce5439a065c42dcba67d9d37b3de096'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:43.256930 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-cb8ba415-1b2a-4332-8643-3915b96d7c1e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dce5439a065c42dcba67d9d37b3de096'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:43.264343 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-cb8ba415-1b2a-4332-8643-3915b96d7c1e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dce5439a065c42dcba67d9d37b3de096'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:43.270264 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1287/2586] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:40:43 2022] DELETE /identity/v3/users/dce5439a065c42dcba67d9d37b3de096 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:40:43.541707 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22054ba0-7427-4098-8884-bfab4823653d admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:43.543557 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-22054ba0-7427-4098-8884-bfab4823653d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:40:43.558594 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22054ba0-7427-4098-8884-bfab4823653d admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:43.617308 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b30a1c2-3b9e-4e15-b64d-ad978c416965 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:43.618022 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b30a1c2-3b9e-4e15-b64d-ad978c416965 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:43.618201 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b30a1c2-3b9e-4e15-b64d-ad978c416965 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:43.618471 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b30a1c2-3b9e-4e15-b64d-ad978c416965 admin admin] PATH_INFO: `/v3/projects/3627d7329b434d4684fda14b7835996e` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:43.619577 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b30a1c2-3b9e-4e15-b64d-ad978c416965 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3627d7329b434d4684fda14b7835996e)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:43.622194 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b30a1c2-3b9e-4e15-b64d-ad978c416965 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:43.668224 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-5b30a1c2-3b9e-4e15-b64d-ad978c416965 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3627d7329b434d4684fda14b7835996e', 'request_id': 'req-5b30a1c2-3b9e-4e15-b64d-ad978c416965'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:43.675364 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-5b30a1c2-3b9e-4e15-b64d-ad978c416965 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3627d7329b434d4684fda14b7835996e', 'request_id': 'req-5b30a1c2-3b9e-4e15-b64d-ad978c416965'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:43.681707 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-5b30a1c2-3b9e-4e15-b64d-ad978c416965 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3627d7329b434d4684fda14b7835996e', 'request_id': 'req-5b30a1c2-3b9e-4e15-b64d-ad978c416965'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:43.688461 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:40:43.693605 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-5b30a1c2-3b9e-4e15-b64d-ad978c416965 admin admin] The token cache is being invalidate because project 3627d7329b434d4684fda14b7835996e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81588) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:40:43.694045 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-5b30a1c2-3b9e-4e15-b64d-ad978c416965 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:43.695042 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-5b30a1c2-3b9e-4e15-b64d-ad978c416965 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:40:43.696653 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1300/2587] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:40:43 2022] DELETE /identity/v3/projects/3627d7329b434d4684fda14b7835996e => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:40:43.702185 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e26ccc7-826a-43db-b761-99da551f62fd None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:43.702684 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e26ccc7-826a-43db-b761-99da551f62fd None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:43.702813 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e26ccc7-826a-43db-b761-99da551f62fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:43.722865 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-4e26ccc7-826a-43db-b761-99da551f62fd None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:40:43.724049 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-4e26ccc7-826a-43db-b761-99da551f62fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:40:43.753299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:40:43.788382 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1288/2588] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:40:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:40:43.797912 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac77acd-74b5-4acc-822b-287aa6c6b86b None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:43.798193 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac77acd-74b5-4acc-822b-287aa6c6b86b None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:43.798463 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac77acd-74b5-4acc-822b-287aa6c6b86b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:43.820017 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-4ac77acd-74b5-4acc-822b-287aa6c6b86b None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:40:43.821283 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-4ac77acd-74b5-4acc-822b-287aa6c6b86b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:40:43.829927 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1301/2589] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:40:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:40:43.834238 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e26ccc7-826a-43db-b761-99da551f62fd None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:43.844514 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e26ccc7-826a-43db-b761-99da551f62fd None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:43.849053 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c999057-9471-4175-aa88-e9606e3f8615 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:43.849679 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c999057-9471-4175-aa88-e9606e3f8615 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:43.849954 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c999057-9471-4175-aa88-e9606e3f8615 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:43.850220 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c999057-9471-4175-aa88-e9606e3f8615 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:43.851339 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c999057-9471-4175-aa88-e9606e3f8615 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 27 01:40:43.851339 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c999057-9471-4175-aa88-e9606e3f8615 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:43.852724 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c999057-9471-4175-aa88-e9606e3f8615 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:43.868568 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1289/2590] 10.222.0.42 () {58 vars in 1153 bytes} [Thu Oct 27 01:40:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 27 01:40:43.869971 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac77acd-74b5-4acc-822b-287aa6c6b86b None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:43.879469 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac77acd-74b5-4acc-822b-287aa6c6b86b None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:43.884196 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfd9f49e-81ee-4c9a-a851-03827e717570 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:43.884795 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd9f49e-81ee-4c9a-a851-03827e717570 admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:43.885094 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd9f49e-81ee-4c9a-a851-03827e717570 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:43.885368 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd9f49e-81ee-4c9a-a851-03827e717570 admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:43.886546 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfd9f49e-81ee-4c9a-a851-03827e717570 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:43.888451 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfd9f49e-81ee-4c9a-a851-03827e717570 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:43.915508 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1302/2591] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:40:43 2022] POST /identity/v3/projects => generated 377 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:40:43.919938 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c999057-9471-4175-aa88-e9606e3f8615 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:43.929816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c999057-9471-4175-aa88-e9606e3f8615 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:43.986742 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0952637-534e-42e7-8a15-7795b3733ecd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:43.987334 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0952637-534e-42e7-8a15-7795b3733ecd admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:43.987607 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0952637-534e-42e7-8a15-7795b3733ecd admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:43.987885 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0952637-534e-42e7-8a15-7795b3733ecd admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:43.988484 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0952637-534e-42e7-8a15-7795b3733ecd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:43.990412 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0952637-534e-42e7-8a15-7795b3733ecd admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:44.010763 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1290/2592] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:40:43 2022] POST /identity/v3/users => generated 289 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:40:44.015393 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfd9f49e-81ee-4c9a-a851-03827e717570 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:44.025638 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfd9f49e-81ee-4c9a-a851-03827e717570 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:44.031142 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e230a1d-48b7-4b7e-9422-88be5976b6d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:44.031784 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e230a1d-48b7-4b7e-9422-88be5976b6d4 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:44.032102 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e230a1d-48b7-4b7e-9422-88be5976b6d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:44.032386 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e230a1d-48b7-4b7e-9422-88be5976b6d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:44.033110 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e230a1d-48b7-4b7e-9422-88be5976b6d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:44.034864 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e230a1d-48b7-4b7e-9422-88be5976b6d4 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:44.044690 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1303/2593] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:40:44 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:40:44.048510 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0952637-534e-42e7-8a15-7795b3733ecd admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:44.059162 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0952637-534e-42e7-8a15-7795b3733ecd admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:44.063702 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2840f25-6566-4eff-bbdb-3cae49a1897d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:44.064287 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2840f25-6566-4eff-bbdb-3cae49a1897d admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:44.064562 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2840f25-6566-4eff-bbdb-3cae49a1897d admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:44.064828 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2840f25-6566-4eff-bbdb-3cae49a1897d admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:44.065522 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2840f25-6566-4eff-bbdb-3cae49a1897d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:44.067086 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2840f25-6566-4eff-bbdb-3cae49a1897d admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:44.078449 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1291/2594] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:40:44 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:40:44.082453 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e230a1d-48b7-4b7e-9422-88be5976b6d4 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:44.093225 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e230a1d-48b7-4b7e-9422-88be5976b6d4 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:44.097812 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f0e46a-8b8c-4061-be68-fd15283ee183 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:44.098448 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f0e46a-8b8c-4061-be68-fd15283ee183 admin admin] REQUEST_METHOD: `PUT` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:44.098722 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f0e46a-8b8c-4061-be68-fd15283ee183 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:44.098991 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f0e46a-8b8c-4061-be68-fd15283ee183 admin admin] PATH_INFO: `/v3/projects/897fac8b218c4476b84ea53d3b79e9a7/users/ed6abf3b558c44f692e3fc95749642a3/roles/f199b79a7178462fa67bb8243943408f` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:44.113626 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f0e46a-8b8c-4061-be68-fd15283ee183 admin admin] RBAC: Authorizing `identity:create_grant(project_id=897fac8b218c4476b84ea53d3b79e9a7, user_id=ed6abf3b558c44f692e3fc95749642a3, role_id=f199b79a7178462fa67bb8243943408f)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:44.116249 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f0e46a-8b8c-4061-be68-fd15283ee183 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:44.129097 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1304/2595] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:40:44 2022] PUT /identity/v3/projects/897fac8b218c4476b84ea53d3b79e9a7/users/ed6abf3b558c44f692e3fc95749642a3/roles/f199b79a7178462fa67bb8243943408f => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:40:44.134008 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db3eafb-4426-46c0-9d86-92de78c261ad None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:44.134291 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db3eafb-4426-46c0-9d86-92de78c261ad None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:44.134562 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db3eafb-4426-46c0-9d86-92de78c261ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:44.160509 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-0db3eafb-4426-46c0-9d86-92de78c261ad None None] MFA Rules not processed for user `fe01809d18f644f9bc72669fc872b53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:40:44.161749 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0db3eafb-4426-46c0-9d86-92de78c261ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:40:44.219637 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1292/2596] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:40:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 27 01:40:44.229144 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f0e46a-8b8c-4061-be68-fd15283ee183 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:44.230919 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-a8f0e46a-8b8c-4061-be68-fd15283ee183 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:40:44.288040 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f0e46a-8b8c-4061-be68-fd15283ee183 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:44.291641 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc8f61d-e7e6-436d-9ce6-ad28466a0c8e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:44.292681 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc8f61d-e7e6-436d-9ce6-ad28466a0c8e service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:44.292962 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc8f61d-e7e6-436d-9ce6-ad28466a0c8e service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:44.293266 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc8f61d-e7e6-436d-9ce6-ad28466a0c8e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:44.301956 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc8f61d-e7e6-436d-9ce6-ad28466a0c8e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:44.303367 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc8f61d-e7e6-436d-9ce6-ad28466a0c8e service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:44.314008 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1305/2597] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:40:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:40:48.357704 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51411c1a-ba9a-47c2-af4b-b0b44edfecaf None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:48.358818 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51411c1a-ba9a-47c2-af4b-b0b44edfecaf None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:48.359002 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51411c1a-ba9a-47c2-af4b-b0b44edfecaf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:48.388892 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-51411c1a-ba9a-47c2-af4b-b0b44edfecaf None None] MFA Rules not processed for user `ed6abf3b558c44f692e3fc95749642a3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:40:48.390179 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-51411c1a-ba9a-47c2-af4b-b0b44edfecaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:40:48.424002 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:40:48.472802 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1293/2598] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:40:48 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:40:48.477136 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc8f61d-e7e6-436d-9ce6-ad28466a0c8e service neutron] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:48.488213 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc8f61d-e7e6-436d-9ce6-ad28466a0c8e service neutron] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:48.492833 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe681bb-7706-4c7b-9564-e9799070579a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:48.493466 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe681bb-7706-4c7b-9564-e9799070579a admin admin] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:48.493742 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe681bb-7706-4c7b-9564-e9799070579a admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:48.494089 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe681bb-7706-4c7b-9564-e9799070579a admin admin] PATH_INFO: `/v3/projects` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:48.494634 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe681bb-7706-4c7b-9564-e9799070579a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:48.498986 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe681bb-7706-4c7b-9564-e9799070579a admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:48.529605 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1306/2599] 10.222.0.42 () {60 vars in 1153 bytes} [Thu Oct 27 01:40:48 2022] POST /identity/v3/projects => generated 377 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:40:48.533339 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51411c1a-ba9a-47c2-af4b-b0b44edfecaf None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:48.545252 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51411c1a-ba9a-47c2-af4b-b0b44edfecaf None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:48.616816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8298b37-5771-4e24-85ce-db91220b285b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:48.617465 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8298b37-5771-4e24-85ce-db91220b285b admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:48.617745 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8298b37-5771-4e24-85ce-db91220b285b admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:48.619125 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8298b37-5771-4e24-85ce-db91220b285b admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:48.619125 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8298b37-5771-4e24-85ce-db91220b285b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:48.621497 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8298b37-5771-4e24-85ce-db91220b285b admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:48.646348 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1294/2600] 10.222.0.42 () {60 vars in 1144 bytes} [Thu Oct 27 01:40:48 2022] POST /identity/v3/users => generated 295 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 27 01:40:48.648625 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe681bb-7706-4c7b-9564-e9799070579a admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:48.659971 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe681bb-7706-4c7b-9564-e9799070579a admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:48.664752 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b74c05-b758-4ce4-8355-965110d0d61b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:48.665502 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b74c05-b758-4ce4-8355-965110d0d61b admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:48.665829 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b74c05-b758-4ce4-8355-965110d0d61b admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:48.666175 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b74c05-b758-4ce4-8355-965110d0d61b admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:48.666852 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75b74c05-b758-4ce4-8355-965110d0d61b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:48.669121 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75b74c05-b758-4ce4-8355-965110d0d61b admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:48.680113 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1307/2601] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:40:48 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:40:48.685578 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8298b37-5771-4e24-85ce-db91220b285b admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:48.695614 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8298b37-5771-4e24-85ce-db91220b285b admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:48.700454 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d90d99-6cf0-41a0-8cf9-076d3001e95e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:48.701139 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d90d99-6cf0-41a0-8cf9-076d3001e95e admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:48.701420 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d90d99-6cf0-41a0-8cf9-076d3001e95e admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:48.701696 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d90d99-6cf0-41a0-8cf9-076d3001e95e admin admin] PATH_INFO: `/v3/projects/0d00e248d64f4e8f958c77a2b208ce53/users/b48cb8af827f487e81e0704028f6ca2d/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:48.718227 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d90d99-6cf0-41a0-8cf9-076d3001e95e admin admin] RBAC: Authorizing `identity:create_grant(project_id=0d00e248d64f4e8f958c77a2b208ce53, user_id=b48cb8af827f487e81e0704028f6ca2d, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:48.720905 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d90d99-6cf0-41a0-8cf9-076d3001e95e admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:48.734342 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1295/2602] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:40:48 2022] PUT /identity/v3/projects/0d00e248d64f4e8f958c77a2b208ce53/users/b48cb8af827f487e81e0704028f6ca2d/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:40:48.738250 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b74c05-b758-4ce4-8355-965110d0d61b admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:48.748044 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b74c05-b758-4ce4-8355-965110d0d61b admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:48.808765 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c173040-1531-4969-886a-968423b18a48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:48.809433 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c173040-1531-4969-886a-968423b18a48 admin admin] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:48.809713 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c173040-1531-4969-886a-968423b18a48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:48.809982 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c173040-1531-4969-886a-968423b18a48 admin admin] PATH_INFO: `/v3/roles` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:48.810628 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c173040-1531-4969-886a-968423b18a48 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:48.812219 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c173040-1531-4969-886a-968423b18a48 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:48.818323 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:40:48.823782 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1308/2603] 10.222.0.42 () {58 vars in 1122 bytes} [Thu Oct 27 01:40:48 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 27 01:40:48.827940 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d90d99-6cf0-41a0-8cf9-076d3001e95e admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:48.838217 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d90d99-6cf0-41a0-8cf9-076d3001e95e admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:48.843488 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5932e6-c431-48d0-83d5-7dcd5f7330e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:48.844132 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5932e6-c431-48d0-83d5-7dcd5f7330e0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:48.844413 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5932e6-c431-48d0-83d5-7dcd5f7330e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:48.844694 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5932e6-c431-48d0-83d5-7dcd5f7330e0 admin admin] PATH_INFO: `/v3/projects/0d00e248d64f4e8f958c77a2b208ce53/users/b48cb8af827f487e81e0704028f6ca2d/roles/414286927fdd44688107ff5e10235acf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:48.846674 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5932e6-c431-48d0-83d5-7dcd5f7330e0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0d00e248d64f4e8f958c77a2b208ce53, user_id=b48cb8af827f487e81e0704028f6ca2d, role_id=414286927fdd44688107ff5e10235acf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:48.849350 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5932e6-c431-48d0-83d5-7dcd5f7330e0 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:48.860274 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1296/2604] 10.222.0.42 () {60 vars in 1483 bytes} [Thu Oct 27 01:40:48 2022] PUT /identity/v3/projects/0d00e248d64f4e8f958c77a2b208ce53/users/b48cb8af827f487e81e0704028f6ca2d/roles/414286927fdd44688107ff5e10235acf => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:40:53.346846 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c015d53c-ec59-4978-867d-3ef7ea60715b None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:53.347327 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c015d53c-ec59-4978-867d-3ef7ea60715b None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:53.347431 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c015d53c-ec59-4978-867d-3ef7ea60715b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:53.395966 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-c015d53c-ec59-4978-867d-3ef7ea60715b None None] MFA Rules not processed for user `b48cb8af827f487e81e0704028f6ca2d`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:40:53.397466 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-c015d53c-ec59-4978-867d-3ef7ea60715b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:40:53.472276 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1309/2605] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:40:53 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:40:53.493169 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5932e6-c431-48d0-83d5-7dcd5f7330e0 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:40:53.495617 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-cb5932e6-c431-48d0-83d5-7dcd5f7330e0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:40:53.576868 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5932e6-c431-48d0-83d5-7dcd5f7330e0 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:40:53.582871 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b16bb36-aa76-48bf-9d1f-200978c38b4b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:40:53.583555 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b16bb36-aa76-48bf-9d1f-200978c38b4b service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:40:53.583828 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b16bb36-aa76-48bf-9d1f-200978c38b4b service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:40:53.584097 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b16bb36-aa76-48bf-9d1f-200978c38b4b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:40:53.603448 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:40:53.647412 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b16bb36-aa76-48bf-9d1f-200978c38b4b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:40:53.649046 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b16bb36-aa76-48bf-9d1f-200978c38b4b service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:40:53.660465 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1297/2606] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:40:53 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:41:03.469073 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=34 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:41:03.655643 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=19 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:41:14.226075 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1077556-7337-4fd8-b986-6e190d9668b3 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:41:14.226581 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1077556-7337-4fd8-b986-6e190d9668b3 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:41:14.226715 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1077556-7337-4fd8-b986-6e190d9668b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:41:14.245326 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-d1077556-7337-4fd8-b986-6e190d9668b3 None None] MFA Rules not processed for user `ed6abf3b558c44f692e3fc95749642a3`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:41:14.246523 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-d1077556-7337-4fd8-b986-6e190d9668b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:41:14.254004 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1310/2607] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:41:14 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:41:14.575314 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b16bb36-aa76-48bf-9d1f-200978c38b4b service nova] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:41:14.576893 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-8b16bb36-aa76-48bf-9d1f-200978c38b4b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:41:14.636101 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b16bb36-aa76-48bf-9d1f-200978c38b4b service nova] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:41:14.639640 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d5c0f9-c633-4813-87b2-71a19d105d34 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:41:14.640439 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d5c0f9-c633-4813-87b2-71a19d105d34 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:41:14.640868 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d5c0f9-c633-4813-87b2-71a19d105d34 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:41:14.641229 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d5c0f9-c633-4813-87b2-71a19d105d34 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:41:14.643015 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-15d5c0f9-c633-4813-87b2-71a19d105d34 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:41:14.652793 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15d5c0f9-c633-4813-87b2-71a19d105d34 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:41:14.654290 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15d5c0f9-c633-4813-87b2-71a19d105d34 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:41:14.663215 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1298/2608] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:41:14 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:41:15.111741 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1077556-7337-4fd8-b986-6e190d9668b3 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:41:15.113296 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-d1077556-7337-4fd8-b986-6e190d9668b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:41:15.184953 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1077556-7337-4fd8-b986-6e190d9668b3 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:41:15.189947 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71022b3-1ac9-4382-931e-f0d929fcf1ae service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62051562fc0b4cd08750d2293e71787c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:41:15.190547 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71022b3-1ac9-4382-931e-f0d929fcf1ae service placement] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:41:15.190822 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71022b3-1ac9-4382-931e-f0d929fcf1ae service placement] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:41:15.191092 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71022b3-1ac9-4382-931e-f0d929fcf1ae service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:41:15.192903 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-c71022b3-1ac9-4382-931e-f0d929fcf1ae service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:41:15.205103 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c71022b3-1ac9-4382-931e-f0d929fcf1ae service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:41:15.206686 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c71022b3-1ac9-4382-931e-f0d929fcf1ae service placement] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:41:15.216521 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1311/2609] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:41:15 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:41:17.028959 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d5c0f9-c633-4813-87b2-71a19d105d34 service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:41:17.038645 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d5c0f9-c633-4813-87b2-71a19d105d34 service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:41:17.042806 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4d2c61-8187-4467-87bd-308652c8dd20 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:41:17.043411 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4d2c61-8187-4467-87bd-308652c8dd20 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:41:17.043687 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4d2c61-8187-4467-87bd-308652c8dd20 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:41:17.043956 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4d2c61-8187-4467-87bd-308652c8dd20 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:41:17.045717 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ca4d2c61-8187-4467-87bd-308652c8dd20 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:41:17.056100 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca4d2c61-8187-4467-87bd-308652c8dd20 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:41:17.057856 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca4d2c61-8187-4467-87bd-308652c8dd20 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:41:17.066893 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1299/2610] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:41:17 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:41:17.786348 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71022b3-1ac9-4382-931e-f0d929fcf1ae service placement] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:41:17.788366 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-c71022b3-1ac9-4382-931e-f0d929fcf1ae service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:41:17.809864 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71022b3-1ac9-4382-931e-f0d929fcf1ae service placement] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:41:17.817690 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246ae519-5d95-4d5d-9729-45325f0e4e90 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:41:17.818774 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246ae519-5d95-4d5d-9729-45325f0e4e90 service nova] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:41:17.819339 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246ae519-5d95-4d5d-9729-45325f0e4e90 service nova] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:41:17.819790 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246ae519-5d95-4d5d-9729-45325f0e4e90 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:41:17.822384 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-246ae519-5d95-4d5d-9729-45325f0e4e90 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:41:17.838006 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-246ae519-5d95-4d5d-9729-45325f0e4e90 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:41:17.839633 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-246ae519-5d95-4d5d-9729-45325f0e4e90 service nova] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:41:17.851314 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1312/2611] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:41:17 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:41:27.063659 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=32 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:41:27.846687 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=41 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:41:29.590414 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4d2c61-8187-4467-87bd-308652c8dd20 service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:41:29.631816 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4d2c61-8187-4467-87bd-308652c8dd20 service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:41:29.637882 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9eeb3f-70d2-48e7-b70a-beefb3a6d481 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:41:29.638514 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9eeb3f-70d2-48e7-b70a-beefb3a6d481 service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:41:29.638789 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9eeb3f-70d2-48e7-b70a-beefb3a6d481 service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:41:29.639056 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9eeb3f-70d2-48e7-b70a-beefb3a6d481 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:41:29.641854 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-2d9eeb3f-70d2-48e7-b70a-beefb3a6d481 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:41:29.654286 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d9eeb3f-70d2-48e7-b70a-beefb3a6d481 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:41:29.655869 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d9eeb3f-70d2-48e7-b70a-beefb3a6d481 service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:41:29.667472 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1300/2612] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:41:29 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 78 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:41:33.336477 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246ae519-5d95-4d5d-9729-45325f0e4e90 service nova] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:41:33.346444 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246ae519-5d95-4d5d-9729-45325f0e4e90 service nova] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:41:33.349968 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7aeaaf5-ba8c-49c8-bf0e-d8b72a1eca6b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:41:33.350546 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7aeaaf5-ba8c-49c8-bf0e-d8b72a1eca6b service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:41:33.350815 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7aeaaf5-ba8c-49c8-bf0e-d8b72a1eca6b service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:41:33.351086 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7aeaaf5-ba8c-49c8-bf0e-d8b72a1eca6b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:41:33.360062 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7aeaaf5-ba8c-49c8-bf0e-d8b72a1eca6b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:41:33.361291 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7aeaaf5-ba8c-49c8-bf0e-d8b72a1eca6b service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:41:33.370332 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1313/2613] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:41:33 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:41:39.663554 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=6 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:41:43.366291 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=6 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:43:16.601646 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9eeb3f-70d2-48e7-b70a-beefb3a6d481 service nova] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:43:16.603060 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-2d9eeb3f-70d2-48e7-b70a-beefb3a6d481 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:43:16.687656 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9eeb3f-70d2-48e7-b70a-beefb3a6d481 service nova] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:43:16.692644 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-372740f8-9ea0-4852-94ea-05484f06b615 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62051562fc0b4cd08750d2293e71787c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:43:16.693273 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372740f8-9ea0-4852-94ea-05484f06b615 service placement] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:43:16.694189 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372740f8-9ea0-4852-94ea-05484f06b615 service placement] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:43:16.694458 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372740f8-9ea0-4852-94ea-05484f06b615 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:43:16.696152 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-372740f8-9ea0-4852-94ea-05484f06b615 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:43:16.707255 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-372740f8-9ea0-4852-94ea-05484f06b615 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:43:16.708826 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-372740f8-9ea0-4852-94ea-05484f06b615 service placement] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:43:16.718528 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1301/2614] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:43:16 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:43:26.714315 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=37 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:43:53.652998 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4246dcc4-f080-42ea-9f23-af3c318df95b None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:43:53.653521 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4246dcc4-f080-42ea-9f23-af3c318df95b None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:43:53.653749 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4246dcc4-f080-42ea-9f23-af3c318df95b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:43:53.673151 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-4246dcc4-f080-42ea-9f23-af3c318df95b None None] MFA Rules not processed for user `ed6abf3b558c44f692e3fc95749642a3`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:43:53.674330 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-4246dcc4-f080-42ea-9f23-af3c318df95b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:43:53.681107 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1314/2615] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:43:53 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:44:03.668162 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=3 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:46:06.296133 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-372740f8-9ea0-4852-94ea-05484f06b615 service placement] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:46:06.297578 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-372740f8-9ea0-4852-94ea-05484f06b615 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:46:06.375564 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-372740f8-9ea0-4852-94ea-05484f06b615 service placement] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:46:06.379143 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f7ad04-ca29-45b5-9203-b93eaeee64ba service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:46:06.379730 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f7ad04-ca29-45b5-9203-b93eaeee64ba service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:46:06.380004 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f7ad04-ca29-45b5-9203-b93eaeee64ba service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:46:06.380784 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f7ad04-ca29-45b5-9203-b93eaeee64ba service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:46:06.388865 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73f7ad04-ca29-45b5-9203-b93eaeee64ba service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:46:06.390303 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73f7ad04-ca29-45b5-9203-b93eaeee64ba service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:46:06.399476 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1302/2616] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:46:06 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:46:16.394891 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=35 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:46:17.446978 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8fbe0f-7709-448e-aa4c-25237bcc7881 None None] REQUEST_METHOD: `POST` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:46:17.448259 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8fbe0f-7709-448e-aa4c-25237bcc7881 None None] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:46:17.448259 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8fbe0f-7709-448e-aa4c-25237bcc7881 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:46:17.469014 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.auth.core [None req-bd8fbe0f-7709-448e-aa4c-25237bcc7881 None None] MFA Rules not processed for user `ed6abf3b558c44f692e3fc95749642a3`. Rule list: `[]` (Enabled: `True`). {{(pid=81588) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 27 01:46:17.470250 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-bd8fbe0f-7709-448e-aa4c-25237bcc7881 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:46:17.477149 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1315/2617] 10.222.0.42 () {58 vars in 958 bytes} [Thu Oct 27 01:46:17 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 27 01:46:17.521183 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f7ad04-ca29-45b5-9203-b93eaeee64ba service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:46:17.531072 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f7ad04-ca29-45b5-9203-b93eaeee64ba service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:46:17.534839 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948fa6d0-d791-4a12-91d2-7dbdd3285453 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd35a3d40a1924064859562a72bab0f6d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:46:17.535456 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948fa6d0-d791-4a12-91d2-7dbdd3285453 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:46:17.535751 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948fa6d0-d791-4a12-91d2-7dbdd3285453 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:46:17.536042 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948fa6d0-d791-4a12-91d2-7dbdd3285453 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:46:17.537810 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-948fa6d0-d791-4a12-91d2-7dbdd3285453 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:46:17.571436 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948fa6d0-d791-4a12-91d2-7dbdd3285453 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:46:17.573491 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948fa6d0-d791-4a12-91d2-7dbdd3285453 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:46:17.594284 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1303/2618] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:46:17 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 74 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:46:17.881878 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd8fbe0f-7709-448e-aa4c-25237bcc7881 None None] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:46:17.901895 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd8fbe0f-7709-448e-aa4c-25237bcc7881 None None] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:46:17.905531 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f4d8a9-a752-4484-b11d-18cfadcc11b1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:46:17.906132 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f4d8a9-a752-4484-b11d-18cfadcc11b1 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:46:17.906435 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f4d8a9-a752-4484-b11d-18cfadcc11b1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:46:17.906708 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f4d8a9-a752-4484-b11d-18cfadcc11b1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:46:17.914031 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f4d8a9-a752-4484-b11d-18cfadcc11b1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:46:17.915456 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f4d8a9-a752-4484-b11d-18cfadcc11b1 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:46:17.923645 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1316/2619] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:46:17 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:46:18.378416 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948fa6d0-d791-4a12-91d2-7dbdd3285453 service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:46:18.391164 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948fa6d0-d791-4a12-91d2-7dbdd3285453 service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:46:18.396267 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f4d8a9-a752-4484-b11d-18cfadcc11b1 service neutron] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:46:18.396422 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdaad739-972b-45ad-be74-f564bd287403 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62051562fc0b4cd08750d2293e71787c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:46:18.396911 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdaad739-972b-45ad-be74-f564bd287403 service placement] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:46:18.397224 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdaad739-972b-45ad-be74-f564bd287403 service placement] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:46:18.397496 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdaad739-972b-45ad-be74-f564bd287403 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:46:18.405673 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f4d8a9-a752-4484-b11d-18cfadcc11b1 service neutron] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:46:18.406875 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdaad739-972b-45ad-be74-f564bd287403 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:46:18.408447 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdaad739-972b-45ad-be74-f564bd287403 service placement] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:46:18.409331 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0261db30-c479-4279-8c20-dad217d0effc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:46:18.409925 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0261db30-c479-4279-8c20-dad217d0effc service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:46:18.410200 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0261db30-c479-4279-8c20-dad217d0effc service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:46:18.410469 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0261db30-c479-4279-8c20-dad217d0effc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:46:18.412169 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-0261db30-c479-4279-8c20-dad217d0effc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:46:18.418393 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1304/2620] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:46:18 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:46:18.426037 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0261db30-c479-4279-8c20-dad217d0effc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:46:18.427447 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0261db30-c479-4279-8c20-dad217d0effc service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:46:18.439553 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1317/2621] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:46:18 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 27 01:46:20.550285 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdaad739-972b-45ad-be74-f564bd287403 service placement] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:46:20.562996 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdaad739-972b-45ad-be74-f564bd287403 service placement] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:46:20.567567 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6e128f-cc20-4b16-93ac-3a172b925321 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:46:20.568566 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6e128f-cc20-4b16-93ac-3a172b925321 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:46:20.569046 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6e128f-cc20-4b16-93ac-3a172b925321 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:46:20.569533 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6e128f-cc20-4b16-93ac-3a172b925321 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:46:20.580150 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6e128f-cc20-4b16-93ac-3a172b925321 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:46:20.581634 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6e128f-cc20-4b16-93ac-3a172b925321 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:46:20.593567 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1305/2622] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:46:20 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:46:21.320939 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0261db30-c479-4279-8c20-dad217d0effc service neutron] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:46:21.333256 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0261db30-c479-4279-8c20-dad217d0effc service neutron] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:46:21.339228 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69330bb-d5b8-417a-b89f-e391d2607660 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:46:21.339833 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69330bb-d5b8-417a-b89f-e391d2607660 service nova] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:46:21.340114 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69330bb-d5b8-417a-b89f-e391d2607660 service nova] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:46:21.340387 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69330bb-d5b8-417a-b89f-e391d2607660 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:46:21.349847 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69330bb-d5b8-417a-b89f-e391d2607660 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:46:21.351428 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69330bb-d5b8-417a-b89f-e391d2607660 service nova] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:46:21.363291 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1318/2623] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:46:21 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:46:30.586600 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=30 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:46:31.358413 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=25 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:46:33.416139 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6e128f-cc20-4b16-93ac-3a172b925321 service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:46:33.430678 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6e128f-cc20-4b16-93ac-3a172b925321 service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:46:33.437000 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ece423-ea77-4e35-87c3-ba80731d5962 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:46:33.437621 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ece423-ea77-4e35-87c3-ba80731d5962 service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:46:33.437924 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ece423-ea77-4e35-87c3-ba80731d5962 service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:46:33.438170 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ece423-ea77-4e35-87c3-ba80731d5962 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:46:33.447819 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ece423-ea77-4e35-87c3-ba80731d5962 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:46:33.449611 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ece423-ea77-4e35-87c3-ba80731d5962 service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:46:33.460278 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1306/2624] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:46:33 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:46:37.550189 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69330bb-d5b8-417a-b89f-e391d2607660 service nova] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:46:37.559895 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69330bb-d5b8-417a-b89f-e391d2607660 service nova] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:46:37.563441 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c103b2d1-edad-403d-aa38-a0cb0210b8c7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:46:37.564029 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c103b2d1-edad-403d-aa38-a0cb0210b8c7 service neutron] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:46:37.564328 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c103b2d1-edad-403d-aa38-a0cb0210b8c7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:46:37.564651 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c103b2d1-edad-403d-aa38-a0cb0210b8c7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:46:37.573418 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c103b2d1-edad-403d-aa38-a0cb0210b8c7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:46:37.574811 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c103b2d1-edad-403d-aa38-a0cb0210b8c7 service neutron] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:46:37.583944 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1319/2625] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:46:37 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 27 01:46:43.456295 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=6 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:46:47.579864 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=6 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:48:20.614133 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ece423-ea77-4e35-87c3-ba80731d5962 service nova] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:48:20.629495 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ece423-ea77-4e35-87c3-ba80731d5962 service nova] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:48:20.634597 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c251e41-84cf-403b-8e60-45d1eaa7736b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62051562fc0b4cd08750d2293e71787c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:48:20.635184 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c251e41-84cf-403b-8e60-45d1eaa7736b service placement] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:48:20.635459 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c251e41-84cf-403b-8e60-45d1eaa7736b service placement] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:48:20.635726 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c251e41-84cf-403b-8e60-45d1eaa7736b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:48:20.646989 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c251e41-84cf-403b-8e60-45d1eaa7736b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:48:20.648878 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c251e41-84cf-403b-8e60-45d1eaa7736b service placement] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:48:20.667798 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1307/2626] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:48:20 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:48:30.661899 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=6 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:48:32.192608 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c103b2d1-edad-403d-aa38-a0cb0210b8c7 service neutron] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:48:32.207213 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c103b2d1-edad-403d-aa38-a0cb0210b8c7 service neutron] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:48:32.212787 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691c84b9-6e19-4329-9cff-90bc01843801 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '51033d56f85f4282a49ded6338440677', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:48:32.213428 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691c84b9-6e19-4329-9cff-90bc01843801 service nova] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:48:32.213710 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691c84b9-6e19-4329-9cff-90bc01843801 service nova] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:48:32.214541 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691c84b9-6e19-4329-9cff-90bc01843801 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:48:32.215870 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-691c84b9-6e19-4329-9cff-90bc01843801 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:48:32.232788 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691c84b9-6e19-4329-9cff-90bc01843801 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:48:32.234480 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691c84b9-6e19-4329-9cff-90bc01843801 service nova] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:48:32.246140 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1320/2627] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:48:32 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:48:42.240880 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=6 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:48:42.316856 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c251e41-84cf-403b-8e60-45d1eaa7736b service placement] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:48:42.327282 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c251e41-84cf-403b-8e60-45d1eaa7736b service placement] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:48:42.330769 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd96ac0d-2d19-4498-b472-3791044cc04d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4072417990c94b0488871332d28224ba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:48:42.331344 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd96ac0d-2d19-4498-b472-3791044cc04d service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:48:42.331617 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd96ac0d-2d19-4498-b472-3791044cc04d service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:48:42.331884 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd96ac0d-2d19-4498-b472-3791044cc04d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:48:42.402002 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd96ac0d-2d19-4498-b472-3791044cc04d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:48:42.403429 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd96ac0d-2d19-4498-b472-3791044cc04d service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:48:42.422606 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1308/2628] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:48:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:48:45.558473 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691c84b9-6e19-4329-9cff-90bc01843801 service nova] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:48:45.571476 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691c84b9-6e19-4329-9cff-90bc01843801 service nova] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:48:45.583348 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d0e222-d084-4aeb-a71f-6d81ba5ae139 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62051562fc0b4cd08750d2293e71787c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:48:45.583348 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d0e222-d084-4aeb-a71f-6d81ba5ae139 service placement] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:48:45.583348 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d0e222-d084-4aeb-a71f-6d81ba5ae139 service placement] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:48:45.583348 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d0e222-d084-4aeb-a71f-6d81ba5ae139 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:48:45.586186 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-52d0e222-d084-4aeb-a71f-6d81ba5ae139 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:48:45.600836 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52d0e222-d084-4aeb-a71f-6d81ba5ae139 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:48:45.602824 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52d0e222-d084-4aeb-a71f-6d81ba5ae139 service placement] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:48:45.615856 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1321/2629] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:48:45 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 58 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:48:48.470889 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd96ac0d-2d19-4498-b472-3791044cc04d service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:48:48.488659 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd96ac0d-2d19-4498-b472-3791044cc04d service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:48:48.499023 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc88b30-72d6-48ca-85f5-e642f656f4d3 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62051562fc0b4cd08750d2293e71787c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:48:48.499791 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc88b30-72d6-48ca-85f5-e642f656f4d3 service placement] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:48:48.500175 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc88b30-72d6-48ca-85f5-e642f656f4d3 service placement] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:48:48.500550 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc88b30-72d6-48ca-85f5-e642f656f4d3 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:48:48.504246 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-2cc88b30-72d6-48ca-85f5-e642f656f4d3 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:48:48.528915 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc88b30-72d6-48ca-85f5-e642f656f4d3 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:48:48.534000 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc88b30-72d6-48ca-85f5-e642f656f4d3 service placement] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:48:48.548296 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1309/2630] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:48:48 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 78 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:48:48.550000 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d0e222-d084-4aeb-a71f-6d81ba5ae139 service placement] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:48:48.573623 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d0e222-d084-4aeb-a71f-6d81ba5ae139 service placement] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:48:48.577923 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a48466-9e82-4349-af3c-6a481fc29db8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:48:48.579085 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a48466-9e82-4349-af3c-6a481fc29db8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:48:48.579085 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a48466-9e82-4349-af3c-6a481fc29db8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:48:48.579344 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a48466-9e82-4349-af3c-6a481fc29db8 admin admin] PATH_INFO: `/v3/users/ed6abf3b558c44f692e3fc95749642a3` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:48:48.580596 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0a48466-9e82-4349-af3c-6a481fc29db8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ed6abf3b558c44f692e3fc95749642a3)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:48:48.582600 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0a48466-9e82-4349-af3c-6a481fc29db8 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:48:48.634649 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-a0a48466-9e82-4349-af3c-6a481fc29db8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ed6abf3b558c44f692e3fc95749642a3'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:48:48.642231 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-a0a48466-9e82-4349-af3c-6a481fc29db8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ed6abf3b558c44f692e3fc95749642a3'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:48:48.655853 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-a0a48466-9e82-4349-af3c-6a481fc29db8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ed6abf3b558c44f692e3fc95749642a3'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:48:48.656847 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-a0a48466-9e82-4349-af3c-6a481fc29db8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ed6abf3b558c44f692e3fc95749642a3'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:48:48.657472 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-a0a48466-9e82-4349-af3c-6a481fc29db8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ed6abf3b558c44f692e3fc95749642a3'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:48:48.669982 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1322/2631] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:48:48 2022] DELETE /identity/v3/users/ed6abf3b558c44f692e3fc95749642a3 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:48:48.843071 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc88b30-72d6-48ca-85f5-e642f656f4d3 service placement] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:48:48.856434 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc88b30-72d6-48ca-85f5-e642f656f4d3 service placement] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:48:48.913615 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5106d9b-0640-4d9b-827d-4bd0829bf8b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:48:48.914236 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5106d9b-0640-4d9b-827d-4bd0829bf8b8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:48:48.914513 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5106d9b-0640-4d9b-827d-4bd0829bf8b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:48:48.914785 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5106d9b-0640-4d9b-827d-4bd0829bf8b8 admin admin] PATH_INFO: `/v3/projects/897fac8b218c4476b84ea53d3b79e9a7` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:48:48.915891 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5106d9b-0640-4d9b-827d-4bd0829bf8b8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=897fac8b218c4476b84ea53d3b79e9a7)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:48:48.918525 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5106d9b-0640-4d9b-827d-4bd0829bf8b8 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:48:48.963694 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f5106d9b-0640-4d9b-827d-4bd0829bf8b8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '897fac8b218c4476b84ea53d3b79e9a7', 'request_id': 'req-f5106d9b-0640-4d9b-827d-4bd0829bf8b8'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:48:48.970040 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f5106d9b-0640-4d9b-827d-4bd0829bf8b8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '897fac8b218c4476b84ea53d3b79e9a7', 'request_id': 'req-f5106d9b-0640-4d9b-827d-4bd0829bf8b8'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:48:48.975714 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f5106d9b-0640-4d9b-827d-4bd0829bf8b8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '897fac8b218c4476b84ea53d3b79e9a7', 'request_id': 'req-f5106d9b-0640-4d9b-827d-4bd0829bf8b8'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:48:48.976760 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:48:49.001945 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f5106d9b-0640-4d9b-827d-4bd0829bf8b8 admin admin] The token cache is being invalidate because project 897fac8b218c4476b84ea53d3b79e9a7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:48:49.002230 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f5106d9b-0640-4d9b-827d-4bd0829bf8b8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:48:49.002849 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f5106d9b-0640-4d9b-827d-4bd0829bf8b8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:48:49.004326 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1310/2632] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:48:48 2022] DELETE /identity/v3/projects/897fac8b218c4476b84ea53d3b79e9a7 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:48:49.008667 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a48466-9e82-4349-af3c-6a481fc29db8 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:48:49.010113 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-a0a48466-9e82-4349-af3c-6a481fc29db8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:48:49.023137 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a48466-9e82-4349-af3c-6a481fc29db8 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:48:49.078949 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-277bce80-18fd-42c4-ad6d-939b3a83fcb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:48:49.079565 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277bce80-18fd-42c4-ad6d-939b3a83fcb8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:48:49.079841 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277bce80-18fd-42c4-ad6d-939b3a83fcb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:48:49.080111 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277bce80-18fd-42c4-ad6d-939b3a83fcb8 admin admin] PATH_INFO: `/v3/users/b48cb8af827f487e81e0704028f6ca2d` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:48:49.081250 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-277bce80-18fd-42c4-ad6d-939b3a83fcb8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b48cb8af827f487e81e0704028f6ca2d)` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:48:49.083148 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-277bce80-18fd-42c4-ad6d-939b3a83fcb8 admin admin] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:48:49.128609 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-277bce80-18fd-42c4-ad6d-939b3a83fcb8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b48cb8af827f487e81e0704028f6ca2d'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:48:49.136436 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-277bce80-18fd-42c4-ad6d-939b3a83fcb8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b48cb8af827f487e81e0704028f6ca2d'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:48:49.141539 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:48:49.149241 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-277bce80-18fd-42c4-ad6d-939b3a83fcb8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b48cb8af827f487e81e0704028f6ca2d'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:48:49.149982 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-277bce80-18fd-42c4-ad6d-939b3a83fcb8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b48cb8af827f487e81e0704028f6ca2d'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:48:49.150706 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.notifications [None req-277bce80-18fd-42c4-ad6d-939b3a83fcb8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b48cb8af827f487e81e0704028f6ca2d'} {{(pid=81588) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:48:49.163700 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1323/2633] 10.222.0.42 () {58 vars in 1224 bytes} [Thu Oct 27 01:48:49 2022] DELETE /identity/v3/users/b48cb8af827f487e81e0704028f6ca2d => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:48:49.331725 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5106d9b-0640-4d9b-827d-4bd0829bf8b8 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:48:49.334694 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f5106d9b-0640-4d9b-827d-4bd0829bf8b8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:48:49.346384 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5106d9b-0640-4d9b-827d-4bd0829bf8b8 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:48:49.403623 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e6f308-a84c-49ff-bbba-223526674c16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fe01809d18f644f9bc72669fc872b53d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '062e383759c34ca3b52c150dd337a6ba', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:48:49.404213 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e6f308-a84c-49ff-bbba-223526674c16 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:48:49.404498 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e6f308-a84c-49ff-bbba-223526674c16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:48:49.404769 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e6f308-a84c-49ff-bbba-223526674c16 admin admin] PATH_INFO: `/v3/projects/0d00e248d64f4e8f958c77a2b208ce53` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:48:49.405926 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e6f308-a84c-49ff-bbba-223526674c16 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0d00e248d64f4e8f958c77a2b208ce53)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:48:49.407882 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e6f308-a84c-49ff-bbba-223526674c16 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:48:49.454322 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-61e6f308-a84c-49ff-bbba-223526674c16 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0d00e248d64f4e8f958c77a2b208ce53', 'request_id': 'req-61e6f308-a84c-49ff-bbba-223526674c16'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:48:49.461538 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-61e6f308-a84c-49ff-bbba-223526674c16 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0d00e248d64f4e8f958c77a2b208ce53', 'request_id': 'req-61e6f308-a84c-49ff-bbba-223526674c16'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:48:49.467299 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-61e6f308-a84c-49ff-bbba-223526674c16 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0d00e248d64f4e8f958c77a2b208ce53', 'request_id': 'req-61e6f308-a84c-49ff-bbba-223526674c16'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:48:49.478078 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-61e6f308-a84c-49ff-bbba-223526674c16 admin admin] The token cache is being invalidate because project 0d00e248d64f4e8f958c77a2b208ce53 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 27 01:48:49.478674 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-61e6f308-a84c-49ff-bbba-223526674c16 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:48:49.479201 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-61e6f308-a84c-49ff-bbba-223526674c16 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 27 01:48:49.480703 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1311/2634] 10.222.0.42 () {58 vars in 1233 bytes} [Thu Oct 27 01:48:49 2022] DELETE /identity/v3/projects/0d00e248d64f4e8f958c77a2b208ce53 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 27 01:48:59.160181 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=9,keystone:DELETE=1 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:48:59.475072 dv-cinder-815582-19-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=59,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 27 01:53:24.603702 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-277bce80-18fd-42c4-ad6d-939b3a83fcb8 admin admin] Authenticating user token {{(pid=81588) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 27 01:53:24.605130 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-277bce80-18fd-42c4-ad6d-939b3a83fcb8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:53:24.742902 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-277bce80-18fd-42c4-ad6d-939b3a83fcb8 admin admin] Validating token access rules against request {{(pid=81588) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 27 01:53:24.800326 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bfc25d5-55da-4151-bc7a-73729eb62723 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62051562fc0b4cd08750d2293e71787c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '23aad848d47c4d27ad4c138ff869e40f', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81588) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 27 01:53:24.800957 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bfc25d5-55da-4151-bc7a-73729eb62723 service placement] REQUEST_METHOD: `GET` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 27 01:53:24.801339 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bfc25d5-55da-4151-bc7a-73729eb62723 service placement] SCRIPT_NAME: `/identity` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 27 01:53:24.801625 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bfc25d5-55da-4151-bc7a-73729eb62723 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81588) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 27 01:53:24.803479 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.fernet_utils [None req-8bfc25d5-55da-4151-bc7a-73729eb62723 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81588) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 27 01:53:24.815730 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bfc25d5-55da-4151-bc7a-73729eb62723 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 27 01:53:24.817536 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bfc25d5-55da-4151-bc7a-73729eb62723 service placement] RBAC: Authorization granted {{(pid=81588) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 27 01:53:24.827458 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: [pid: 81588|app: 0|req: 1324/2635] 10.222.0.42 () {60 vars in 1365 bytes} [Thu Oct 27 01:53:24 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 224 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 27 01:53:34.823691 dv-cinder-815582-19-iscsi devstack@keystone.service[81588]: DEBUG dbcounter [-] [81588] Writing DB stats keystone:SELECT=89 {{(pid=81588) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}